If you fetch the data from any ContentProvider in your application, the most likely scenarios are:

1. You’re absolutely clueless and fetch the data on UI thread.
2. You’re using an AsyncTask and:
   • Your app crashes on screen orientation change.
   • You block the screen orientation, because you googled this “solution” on StackOverflow.
   • You wrote error prone boilerplate code to detach and reattach the AsyncTask from the Activity.
3. You are using CursorLoader.

But accessing ContentProvider is probably not the only type of asynchronous operations you perform. You might want to access SharedPreferences, read a file or query a web API. In that case you need Loader<SomeOtherDataThanCursor>, but implementing one correctly is a bit tricky.

I’ll walk you through the entire process of understanding how the Loaders work, implementing a sane base class for your Loaders, implementing CursorLoader with all issues fixed and extending it to allow multiple notification Uris. It’ll be a long post so grab a cup of your favourite beverage.

Loaders 101

Loader should do three things:

1. Load data in background thread.
2. Cache the loaded data, so you won’t reload it after screen orientation change.
3. If applicable, monitor the data source and reload the data when necessary.

The Loader class itself doesn’t provide any mechanism for loading the data in background thread. You either have to implement this yourself, or you can subclass the AsyncTaskLoader. This covers the first point on our requirements list.

The 2nd point is not handled by AsyncTaskLoader. In fact the AsyncTaskLoader is far from being fully functional, for example this perfectly reasonable looking implementation won’t work:

1
2
3
4
5
6
7
8
9
10

public class DasLoader extends AsyncTaskLoader<String> {
  public DasLoader(Context context) {
    super(context);
  }

  @Override
  public String loadInBackground() {
    return "Das";
  }
}

AbstractLoader v1

A good starting point for creating either CursorLoader implementation or LoaderCustom.java from SDK samples. Here’s the common part of these two implementations, which provides all necessary boilerplate for loading and caching the data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

public abstract class AbstractLoader<T> extends AsyncTaskLoader<T> {
  T mResult;

  public AbstractLoader(Context context) {
    super(context);
  }

  @Override
  public void deliverResult(T result) {
    if (isReset()) {
      releaseResources(result);
      return;
    }

    T oldResult = mResult;
    mResult = result;

    if (isStarted()) {
      super.deliverResult(result);
    }

    if (oldResult != result && oldResult != null) {
      releaseResources(oldResult);
    }
  }

  @Override
  public void onCanceled(T result) {
    super.onCanceled(result);
    releaseResources(result);
  }

  @Override
  protected void onReset() {
    super.onReset();

    // Ensure the loader is stopped
    onStopLoading();

    releaseResources(mResult);
    mResult = null;
  }

  @Override
  protected void onStartLoading() {
    if (mResult != null) {
      deliverResult(mResult);
    }
    if (takeContentChanged() || mResult == null) {
      forceLoad();
    }
  }

  @Override
  protected void onStopLoading() {
    cancelLoad();
  }

  protected void releaseResources(T result) {
  }
}

Why this class is not provided by the framework is a mystery to me, but hey, that’s just one more thing you have to know when coding for Android platform. Now you can write your custom Loader like this:

1
2
3
4
5
6
7
8
9
10

public class DasLoader extends AbstractLoader<String> {
  public DasLoader(Context context) {
    super(context);
  }

  @Override
  public String loadInBackground() {
    return "Das";
  }
}

But why do we need all that code? The key to understanding the Loaders is understanding the expected Loader’s behaviour in different states: started, stopped, abandoned and reset. Upon entering each state, the appropriate callback is executed:

• onStartLoading: the Loader was created and should either load the data or return cached data.
• onStopLoading: the Loader should keep the cached data and monitor the data source for changes, but it shouldn’t load the data. This happens for example when users presses home button from your app.
• onAbandoned: someone restarted the Loader. New instance of this Loader was created in onCreateLoader callback in your Fragment/Activity/whatever and loads new data. The abandoned Loader should keep the data until the new Loader loads and delivers it’s data. There is no point of monitoring data source or reloading the data in abandoned Loader – the data will be loaded by the new instance. When new Loader delivers it’s data this Loader will be reset.
• onReset: the data previously loaded by this Loader are no longer used and should be cleaned up. This Loader might be started again, so make sure you clean up also any old state in your Loader implementation.

The AsyncTaskLoader provides additional callback:

• onCancelled: called after data loading when it turns out that this data is no longer needed, for example when the AsyncTask executing your onLoadInBackground was cancelled. In this callback you should take care of releasing resources.

Since the releasing resources should be also performed in onReset callback and in our deliverResults implementation, our AbstractLoader class provides handy releaseResources() callback for closing your Cursors, file handles, etc.

Now let’s walk through our AbstractLoader implementation. When someone starts our Loader using LoaderManager.initLoader(), the onStartLoading is called:

1
2
3
4
5
6
7
8
9
10
11
12
13

  T mResult;

  // ...

  @Override
  protected void onStartLoading() {
    if (mResult != null) {
      deliverResult(mResult);
    }
    if (takeContentChanged() || mResult == null) {
      forceLoad();
    }
  }

We keep the loaded data in mResult member of our AbstractLoader. If we have already loaded the data, we can just deliver the results to Loader clients. If the cache is empty or the Loader was notified about new data available for fetching, we force data reload by calling forceLoad() method. It starts AsyncTask which executes loadInBackground in background thread and the result is passed to deliverResults function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

  @Override
  public void deliverResult(T result) {
    if (isReset()) {
      releaseResources(result);
      return;
    }

    T oldResult = mResult;
    mResult = result;

    if (isStarted()) {
      super.deliverResult(result);
    }

    if (oldResult != result && oldResult != null) {
      releaseResources(oldResult);
    }
  }

A lot of interesting things happen here. First, we check if the loader was put into reset state. In this state all the previous resources were already released, so we just need to take care of newly loaded data. Then we swap the data in cache, call deliverResults in Loader and then release the resources for previously cached data.

When the Fragment or Activity with active Loader is stopped, the Loaders are also put in the stopped state. It means that they should keep the cached data, monitor if this data is still valid, but they should not actively load the data or deliver the results to UI thread. In terms of AsyncTaskLoader it means that any running AsyncTasks should be cancelled:

1
2
3
4

  @Override
  protected void onStopLoading() {
    cancelLoad();
  }

Current implementation of AsyncTaskLoader do not interrupt the active tasks, it only marks that the results of these tasks should not be delivered to the UI thread. However, the results might require some resource releasing, so the onCancelled callback is called:

1
2
3
4
5

  @Override
  public void onCanceled(T result) {
    super.onCanceled(result);
    releaseResources(result);
  }

The last callback we have to implement is onReset:

1
2
3
4
5
6
7
8
9
10

  @Override
  protected void onReset() {
    super.onReset();

    // Ensure the loader is stopped
    onStopLoading();

    releaseResources(mResult);
    mResult = null;
  }

There are two important things here. First, the Loader can be moved to reset state from started state, which means it can still have active AsyncTasks executing loadInBackground. We need to stop them first. Then, as per the specified contract, we have to release the resources and clear the cache.

What about onAbandoned callback? AbstractLoader doesn’t monitor any data source by itself, so this callback doesn’t have to be implemented.

CursorLoader

So how would we implement observing a data source and automatic reloading? Let’s see how would the CursorLoader implementation look like had we used our AbstractLoader as a base class (literally; if you merge MyCursorLoader and AbstractLoader code samples from this post, you’ll get exactly the CursorLoader implementation from support-v4):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public class MyCursorLoader extends AbstractLoader<Cursor> {
  private final ForceLoadContentObserver mObserver;

  public MyCursorLoader(Context context) {
    super(context);
    mObserver = new ForceLoadContentObserver();
  }

  // bunch of setters for uri, projection, selection, etc. Omitted for brevity

  @Override
  public Cursor loadInBackground() {
    Cursor cursor = getContext().getContentResolver().query(mUri, mProjection, mSelection,
        mSelectionArgs, mSortOrder);
    if (cursor != null) {
      // Ensure the cursor window is filled
      cursor.getCount();
      cursor.registerContentObserver(mObserver);
    }
    return cursor;
  }
}

This implementation has two bugs and one kind-of-feature. Let’s start with the last one.

onStartLoading contract specifies, that you should start monitoring the data source upon entering this state. But let’s think what would happen if you had a query that takes 200ms to run and your data source would change every 150ms. The loader would never deliver any data, because every load request would be cancelled in middle of loadInBackground execution by our content observer.

I guess that’s why the Android implementation of CursorLoader registers the observer after the data is loaded. This way the first results are delivered as soon as possible, but for subsequent loads the data is delivered only when data didn’t change during loading. I’m not sure if it was intentional, or this behavior was implemented by accident, but it makes sense to me, so let’s adjust our Loaders contract and implement this behavior in our AbstractLoader.

But if you look closely, the CursorLoader implementation violates even this updated contract. Remember that loadInBackground and deliverResults are executed on separate threads. So what would happen if the data observer is triggered after registerContentObserver call, but before the deliverResults? We’d get exactly the same behavior we’d get had we registered the ContentObserver in onStartLoading – the loader would never deliver it’s data. That’s the first bug.

The second issue with CursorLoader implementation is violation of onAbandon callback contract. If someone calls restartLoader and the content observer is triggered, the abandoned Loader instance will happily reload it’s data just to throw it away.

You can dismiss it as something that would happen only 1% of the time and has negligible impact, and if we were talking about application code, I’d agree with you, but IMO library code that will be used by thousands of developers should be held to a higher standard.

Fixing CursorLoader

Here’s the wrap up of changes in behavior: 1. Register ContentObserver after the first data is delivered, not after the first data is loaded. 2. Unregister ContentObserver in onAbandon.

The first point requires changes to deliverResult method, so it makes sense to modify our AbstractLoader:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

  @Override
  public void deliverResult(T result) {
    if (isReset()) {
      releaseResources(result);
      return;
    }

    T oldResult = mResult;
    mResult = result;

    if (isStarted()) {
      if (oldResult != result) {
        onNewDataDelivered(result);
      }
      super.deliverResult(result);
    }

    if (oldResult != result && oldResult != null) {
      releaseResources(oldResult);
    }
  }

  protected void onNewDataDelivered(T data) {
  }

Our CursorLoader implementation would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

public class MyCursorLoader extends AbstractLoader<Cursor> {
  private final ForceLoadContentObserver mObserver;

  public MyCursorLoader(Context context) {
    super(context);
    mObserver = new ForceLoadContentObserver();
  }

  // bunch of setters for uri, projection, selection, etc. Omitted for brevity

  @Override
  public Cursor loadInBackground() {
    Cursor cursor = getContext().getContentResolver().query(mUri, mProjection, mSelection,
        mSelectionArgs, mSortOrder);
    if (cursor != null) {
      // Ensure the cursor window is filled
      cursor.getCount();
    }
    return cursor;
  }

  @Override
  protected void onNewDataDelivered(Cursor data) {
    super.onNewDataDelivered(data);
    data.registerContentObserver(mObserver);
  }
}

The second part – unregistering observer in onAbandon – is tricky. It’s illegal to call Cursor.unregisterContentObserver with observer that wasn’t registered and the onAbandon can be called when the deliverResults wasn’t called (see AsyncTaskLoader.dispatchOnLoadComplete() implementation). One solution would be keeping the set of Cursors that were registered, but it’s not optimal. Instead, we can create a proxy ContentObserver that can be enabled or disabled:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class DisableableContentObserver extends ContentObserver {
  private final ContentObserver mWrappedObserver;
  private boolean mIsEnabled = true;

  public DisableableContentObserver(ContentObserver wrappedObserver) {
    super(new Handler());
    mWrappedObserver = wrappedObserver;
  }

  @Override
  public void onChange(boolean selfChange) {
    if (mIsEnabled) {
      mWrappedObserver.onChange(selfChange);
    }
  }

  public void setEnabled(boolean isEnabled) {
    mIsEnabled = isEnabled;
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

public class MyCursorLoader extends AbstractLoader<Cursor> {
  private final DisableableContentObserver mObserver;

  public MyCursorLoader(Context context) {
    super(context);
    mObserver = new DisableableContentObserver(new ForceLoadContentObserver());
  }

  // bunch of setters for uri, projection, selection, etc. Omitted for brevity

  @Override
  protected void onStartLoading() {
    mObserver.setEnabled(true);
    super.onStartLoading();
  }

  @Override
  protected void onAbandon() {
    mObserver.setEnabled(false);
  }

  @Override
  protected void onReset() {
    mObserver.setEnabled(false);
    super.onReset();
  }

  @Override
  public Cursor loadInBackground() {
    Cursor cursor = getContext().getContentResolver().query(mUri, mProjection, mSelection,
        mSelectionArgs, mSortOrder);
    if (cursor != null) {
      // Ensure the cursor window is filled
      cursor.getCount();
    }
    return cursor;
  }

  @Override
  protected void onNewDataDelivered(Cursor data) {
    super.onNewDataDelivered(data);
    data.registerContentObserver(mObserver);
  }
}

AbstractObservingLoader

The CursorLoader is a bit special case, because the Cursor itself contains ContentObservable. In most cases however the content observers and loaded data would be completely separated. For these cases it would be useful to have a base class for Loader which registers some ContentObservers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public abstract class AbstractObservingLoader<T> extends AbstractLoader<T> {
  protected final DisableableContentObserver mObserver;
  private boolean mIsRegistered;

  public AbstractObservingLoader(Context context) {
    super(context);
    mObserver = new DisableableContentObserver(new ForceLoadContentObserver());
  }

  @Override
  protected void onStartLoading() {
    mObserver.setEnabled(true);
    super.onStartLoading();
  }

  @Override
  protected void onAbandon() {
    mObserver.setEnabled(false);
    unregisterObserver(mObserver);
    mIsRegistered = false;
  }

  @Override
  protected void onReset() {
    mObserver.setEnabled(false);
    unregisterObserver(mObserver);
    mIsRegistered = false;z
    super.onReset();
  }

  @Override
  protected void onNewDataDelivered(T data) {
    if (!mIsRegistered) {
      mIsRegistered = true;
      registerObserver(mObserver);
    }
  }

  protected abstract void registerObserver(ContentObserver observer);
  protected abstract void unregisterObserver(ContentObserver observer);
}

We need to keep the registered state in our Loader, because the default Observable implementation doesn’t like registering the same observer twice or unregistering not registered observer.

Now we can use this class as a base for a Loader which should be reloaded when one of specified Uris is triggered:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

public class MyCursorLoader extends AbstractObservingLoader<Cursor> {

  public MyCursorLoader(Context context) {
    super(context);
  }

  // bunch of setters for uri, projection, selection, etc. Omitted for brevity

  @Override
  public Cursor loadInBackground() {
    Cursor cursor = getContext().getContentResolver().query(mUri, mProjection, mSelection,
        mSelectionArgs, mSortOrder);
    if (cursor != null) {
      // Ensure the cursor window is filled
      cursor.getCount();
    }
    return cursor;
  }

  @Override
  protected void onNewDataDelivered(Cursor data) {
    super.onNewDataDelivered(data);
    data.registerContentObserver(mObserver);
  }

  @Override
  protected void registerObserver(ContentObserver observer) {
    for (Uri uri : mObservedUris) {
      getContext().getContentResolver().registerContentObserver(uri, true, observer);
    }
  }

  @Override
  protected void unregisterObserver(ContentObserver observer) {
    getContext().getContentResolver().unregisterContentObserver(observer);
  }
}

Conclusions

I think the Loaders aren’t as bad as some people think and say. Four Loader states might seem complicated at first, but if you think about Android Activity lifecycle they make perfect sense and they are something you’d have to implement yourself, with a high probability of mucking things up. The only thing lacking is documentation and sane base classes for extensions, something I hope I delivered through this blog post.

+CommonsWare wrote few weeks ago that he considers Loader to be a failed abstraction, mostly because the interface assumes there is a single object which notifies the Loader about new data. He concluded his post with the following sentence:

In my case, if I am going to have some singleton manager object, with distinct data objects per operation, I am going to use something more flexible than Loader, such as an event bus.

Extending AbstractObservingLoader to load some data from SQLiteDatabase and subscribe to some event bus for model change events should be trivial, and you’d get a lot of things for free – performing loads in background, cancelling loads, caching results, invalidating cache, and so on.

Having said that, Loaders are not solution to every single problem. They are coupled with activity lifecycle, so they are not suitable for long running tasks that should not be interrupted when user navigates away. In these cases the IntentService, or some other Service implementation is a better choice.

The post title raises two questions: “why would you want to use ProGuard in debug builds?” and “why the hell are you still using ant?!”.

The answer for the first question is simple, so let’s start with that one. A while ago we hit the infamous 64k method limit. The most reasonable option is to use ProGuard to remove unused code, especially from the dependencies. For some unreasonable option, see the Jesse Wilson’s comments in the linked bug report.

And why ant instead of Maven, gradle, buck or any other non-antique build tool? We’ve started the project in dark ages (in 2011), when the android Maven plugin was still under heavy development and we encontered several issues with it. Ant builds just worked, so we set it up and nobody had to touch it for several months. At one point we considered switching to gradle, because setting up some annotation processors with ant was a pain in the ass, but again there were some issues with using gradle in our setup at the time (IIRC it was not possible to use local aar dependencies, and the builds were sooooooo slooooooow), so we did not switch.

It all boils down to this: I can spend several hours working on some useful feature or library or I can use this time to switch to another build system. Until the latter option gives me some real benefits, I’d rather work on some features.

Anyways, using ProGuard for IDE builds is easy: go to Project Structure, Facets, and fill appropriate options on ProGuard tab for your project. That’s fine for developers workflow, but not for CI.

To make ProGuard work with ant you need to customize your build.xml. First, if this is your first cusomization, change the version tag in build.xml as recommended:

1

<!-- version-tag: custom -->

Then override -debug-obfuscation-check and -release-obfuscation-check targets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

<target name="-set-proguard-config">
    <condition property="proguard.config" value="proguard.cfg" else="proguard_debug.cfg">
        <isfalse value="${build.is.packaging.debug}"/>
    </condition>
</target>

<target name="-debug-obfuscation-check" depends="-set-proguard-config">
    <!-- enable proguard even in debug mode -->
    <property name="proguard.enabled" value="true"/>
    <echo>proguard.config is ${proguard.config}</echo>

    <!-- Secondary dx input (jar files) is empty since all the jar files will be in the obfuscated jar -->
    <path id="out.dex.jar.input.ref" />
</target>

<target name="-release-obfuscation-check" depends="-set-proguard-config">
    <echo level="info">proguard.config is ${proguard.config}</echo>
    <condition property="proguard.enabled" value="true" else="false">
        <and>
            <isset property="build.is.mode.release" />
            <isset property="proguard.config" />
        </and>
    </condition>
    <if condition="${proguard.enabled}">
        <then>
            <echo level="info">Proguard.config is enabled</echo>
            <!-- Secondary dx input (jar files) is empty since all the
                 jar files will be in the obfuscated jar -->
            <path id="out.dex.jar.input.ref" />
        </then>
    </if>
</target>

Note that I’ve also introduced a -set-proguard-config task to be able to select different configuration for debug and release builds. We do not want to do obfuscation or advanced optimisations in debug, a simple dead code pruning is all we need. Since ant properties are immutable, this means that you HAVE TO remove proguard.config from your ant.properties.

I’m not exactly sure when this happened, but at some point our build times skyrocketed to over 3 minutes. It’s fine for release or CI scripts, but absolutely unnacceptable for developers workflow. Fortunately it was enough to bump the heap size for Proguard. In Android Studio go to Settings, Compiler, Android Compilers and pass -Xmx1024m to ProGuard VM options.

One of the UI patterns that improve lists usability is dividing items into sections. The section might be the first letter of the main text on the list item, date formatted and rounded in a specific way or whatever makes sense for your data.

From the technical point of view you can either add the header view to every list item and show and hide them as needed or create the separate view for header and regular list item and register multiple view types in your Adapter. Both options were described in details by +Cyril Mottier in excellent ListView Tips & Tricks #2: Sectioning Your ListView blog post.

If you choose the second approach, you’ll have to decide what to return from your Adapter’s getItem and getItemId methods for items representing sections. If your sections are not supposed to be clickable, you might implement your Adapter like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

@Override
public Object getItem(int position) {
  return getItemViewType(position) == TYPE_ITEM
      ? mItems[getCursorPosition(position)]
      : null;
}

@Override
public long getItemId(int position) {
  return getItemViewType(position) == TYPE_ITEM
      ? getCursorPosition(position)
      : 0;
}

@Override
public boolean areAllItemsEnabled() {
  return false;
}

@Override
public boolean isEnabled(int position) {
  return getItemViewType(position) == TYPE_ITEM;
}

And your onListItemClickListener like this:

1
2
3
4
5
6
7

@Override
public void onListItemClick(ListView l, View v, int position, long id) {
  super.onListItemClick(l, v, position, id);

  // dummy action which uses Object returned from getItem(position)
  Log.d("DMFM", getListAdapter().getItem(position).toString());
}

If you do so, the Android has a nasty surprise for you:

1
2
3
4
5
6
7
8
9

java.lang.NullPointerException
    at org.chalup.dialmformonkey.app.MainFragment.onListItemClick(MainFragment.java:38)
    at android.app.ListFragment$2.onItemClick(ListFragment.java:160)
    at android.widget.AdapterView.performItemClick(AdapterView.java:298)
    at android.widget.AbsListView.performItemClick(AbsListView.java:1100)
    at android.widget.AbsListView$PerformClick.run(AbsListView.java:2749)
    at android.widget.AbsListView$1.run(AbsListView.java:3423)
    at android.os.Handler.handleCallback(Handler.java:725)
    ...

The only way this can happen is getting null from Adapter.getItem(), but this method will be called only for disabled items, right?

1
2
3
4
5
6
7
8
9
10
11
12
13

@Override
public void onListItemClick(ListView l, View v, int position, long id) {
  super.onListItemClick(l, v, position, id);

  Log.d("DMFM", "Clicked on item " + position + " which is " +
        (getListAdapter().isEnabled(position)
            ? "enabled"
            : "disabled")
  );

  // dummy action which uses Object returned from getItem(position)
  Log.d("DMFM", getListAdapter().getItem(position).toString());
}

Wrong:

1
2
3
4
5
6
7
8
9
10

D/DMFM﹕ Clicked on item 4 which is enabled
D/DMFM﹕ Abondance
D/DMFM﹕ Clicked on item 4 which is enabled
D/DMFM﹕ Abondance
D/DMFM﹕ Clicked on item 31 which is enabled
D/DMFM﹕ Aragon
D/DMFM﹕ Clicked on item 31 which is enabled
D/DMFM﹕ Aragon
D/dalvikvm﹕ GC_CONCURRENT freed 138K, 3% free 8825K/9016K, paused 0ms+0ms, total 3ms
D/DMFM﹕ Clicked on item 28 which is disabled

It’s very difficult to reproduce this error manually, especially if tapping the list item does something more than writing to logcat, but I investigated this issue, because the stack traces above appeared in crash reports on Google Analytics, so several people managed to click exactly wrong area at the wrong time.

I didn’t investigate the issue thoroughly, but it seems there must be some disparity between checking the isEnabled method and getting the item. If I ever dive into ListView code, I’ll definitely write about it. If you want to reproduce or investigate the issue yourself, compile this project and run the monkey:

1

$ adb shell monkey -p org.chalup.dialmformonkey.app -v 500

So what can we do? First option is checking the Adapter.isEnabled() in your onListItemClick listener, which is yet another kind of boilerplate you have to add to your Android code, but it’s super easy to add. The other option is going with the first sectioning approach, i.e. putting section as a part of the clickable list item, but it might not work for your use case (for example adapter with multiple item types).

This week I spend few days analyzing and fixing various issues of app widget in Base CRM application.

This part of our codebase was created over a year ago during one of our internal hackathons and was released soon after that. Most of the times it worked. Every once in a while we received a weird crash report from Google Analytics, but it never caused much trouble. Recently though we received few complaints from customers. I happened to have few hours available for bug hunting, so I took a dive.

The widget is really a simple todo list backed by ContentProvider. The code looks like it was based on the WeatherWidget from SDK samples. What can possibly go wrong?

Issue #1: gazillions of threads started

Take a look at the code of WeatherWidgetProvider:

1
2
3
4
5
6

public WeatherWidgetProvider() {
  // Start the worker thread
  sWorkerThread = new HandlerThread("WeatherWidgetProvider-worker");
  sWorkerThread.start();
  sWorkerQueue = new Handler(sWorkerThread.getLooper());
}

The WeatherWidgetProvider is an AppWidgetProvider implementation, which extends a regular BroadcastReceiver. It means that for every action a new instance of WeatherWidgetProvider is created, and the current implementation spawns new thread which is never closed.

The sample author obviously intended to create only one worker thread – the sWorkerThread is the static – but forgot to do the null check before creating a new thread. So let’s fix it:

1
2
3
4
5
6
7
8

public WeatherWidgetProvider() {
  if (sWorkerThread == null) {
    // Start the worker thread
    sWorkerThread = new HandlerThread("WeatherWidgetProvider-worker");
    sWorkerThread.start();
    sWorkerQueue = new Handler(sWorkerThread.getLooper());
  }
}

Issue #2: no refresh after application update

The widget shows data from the same ContentProvider as the main app, so when the user creates a task inside in the main app and then goes back to homescreen, the task should be displayed on the widget. To achieve this we did the same thing the WeatherWidget sample does – we register the ContentObserver in onEnabled callback of AppWidgetProvider:

1
2
3
4
5
6
7
8
9
10

@Override
public void onEnabled(Context context) {
  final ContentResolver r = context.getContentResolver();
  if (sDataObserver == null) {
    final AppWidgetManager mgr = AppWidgetManager.getInstance(context);
    final ComponentName cn = new ComponentName(context, WeatherWidgetProvider.class);
    sDataObserver = new WeatherDataProviderObserver(mgr, cn, sWorkerQueue);
    r.registerContentObserver(WeatherDataProvider.CONTENT_URI, true, sDataObserver);
  }
}

The onEnabled callback is called when the first instance of the widget is added to homescreen, so the code looks fine. Unfortunately the callback is not called at your process startup. So if your app is updated and the process is restarted, the ContentObserver won’t be registered. The same thing happens if your app crashes or is stopped by the OS to free resources.

To solve this you have to register the ContentObserver in few more places. I have added registration to onCreate callback in RemoteViewsFactory and the onReceive part which handles our custom actions in AppWidgetProvider.

WeatherWidget sample does one more thing wrong: the ContentObserver is never unregistered and the worker thread is never stopped. The correct place to do this is onDisabled callback in AppWidgetProvider.

Issue #3: CursorOutOfBoundsException crash

Ever since we introduced the tasks widget, we’ve occasionally received the crash reports indicating that the RemoteViewsFactory requested elements outside of [0, getCount) range:

1
2
3

05-10 15:22:50.559  13781-13795/org.chalup.widgetfail.widget E/AndroidRuntime﹕ FATAL EXCEPTION: Binder_2
    Process: org.chalup.widgetfail.widget, PID: 13781
    android.database.CursorIndexOutOfBoundsException: Index 1 requested, with a size of 1

The reproduction steps for this issue are quite complicated:

• Tap the task on the widget to mark it was completed. Internally we set the PENDING_DONE flag, so the task is marked as done, but is still displayed on the list, so the user can tap it again and reset the flag.
• Trigger the sync
• SyncAdapter syncs the Task to our backend. The task is marked as DONE in our database, which triggers the ContentObserver registered by the widget.
• ContentObserver triggers onDataSetChanged callback in RemoteViewsFactory, which then calls getCount and getViewAt
• In some rare cases getViewAt with position == result of getCount is called

It looks like some kind of a race condition or another threading issue in Android code which populates the app widgets. I tried synchronizing the RemoteViewsFactory methods, but it didn’t help. The getViewAt have to return a valid RemoteViews, so I fixed it up by returning the loading view when element outside of valid range is requested:

1
2
3
4
5
6
7
8
9
10

@Override
public synchronized RemoteViews getViewAt(int position) {
  if (position >= mCursor.getCount()) {
    return getLoadingView();
  } else {
    mCursor.moveToPosition(position);

    // ...
  }
}

Issue #4: no refresh when “Don’t keep activities” setting is enabled

User can click on the tasks displayed on the widget to go to the edit screen. The activity is closed when user saves or discards changes and the homescreen with the widget is shown again. Changing the task triggers the ContentObserver, the onDataSetChanged is called on all active RemoteViewsFactories, but sometimes other callbacks (getCount, getViewAt, etc.) are not called.

It turns out this happens when the Homescreen activity is recreated because of low memory condition. To easily reproduce this issue you can check the “Don’t keep activities” in developers settings.

I do not have a solution or workaround for this issue unfortunately. I’ll file a bug report and hope for the best.

Recap

There are mutliple issues with the WeatherWidget sample and some issues with the system services responsible for populating app widgets with content. I’ve created a simple project which reproduces the issues #3 and #4 and shows the correct way of registering ContentObserver for your widget. The sources are available on Github.

I’ve pushed new versions of microorm and thneed to Maven Central today. The most notable change for both libraries is dropping the support for Android 2.2 and earlier versions. The same change was applied to all Android libraries open sourced by Base. Why? +Jeff Gilfelt summed it up nicely:

Because it is 2014 https://t.co/UCMaZOB6Sl

— Jeff Gilfelt (@readyState) February 17, 2014

This tweet is a good laugh (and an excellent example of what happens if you limit the discussion to 140 characters), but there are poor souls who might need an answer they can use as an objective argument. For them, here is my take on this one: you should drop support for Froyo because sizeable chunk of Java 1.6 APIs were missing from API level 8. I’m not talking about some dark corners of java packages, I’m talking about stuff like String.isEmpty(), Deque, NavigableSet, IOException’s constructors with cause parameter, and so on.

Your own code can (and should) be checked with Lint, but these methods and classes can also be used by the 3rd party libraries and I’m not aware of any static analysis tool that can help you in this case. So if your app supports Froyo and uses a lot of external dependencies, you’re probably sitting on the NoClassDefFoundError bomb. It might force you to use obsolete versions of libraries, the most notable example of which is Guava – on Froyo you have to use 13.0.1, a 18 months old version.

That’s also the reason why the libraries authors should be the first ones to move on to Android 2.3 and later. If you use obsolete library in your application, you’re hurting only yourself. If you use it as a library dependency, you’re hurting every user of the library.

So move on and bump the minSdkVersion. After all, it’s 2014.

Yesterdays #AndroidDev #Protip explains how to use WakefulBroadcastReceiver utility class and what problem does it solve, but it doesn’t mention a case when using it or manually acquiring WakeLock is essential – using the AlarmManager.

If you’re not familiar with AlarmManager’s API, here is tl;dr of the docs: it allows you to specify the PendingIntent that should be fired at some point, even if your application is in background. The common use cases for using AlarmManager is for example showing a Notification at the specified time or sending some kind of heartbeat to your backend. In both cases, your code performs potentially long running operation (in case of showing notification you might need some content from your local database), so you don’t want to run it in the UI thread. The first thing that comes to mind is to specify an IntentService as a PendingIntent target:

1
2
3
4
5
6
7
8
9
10
11
12
13

PendingIntent intent = PendingIntent.getService(
  context,
  0,
  new Intent(context, MyIntentService.class),
  PendingIntent.FLAG_UPDATE_CURRENT
);

AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
alarmManager.set(
  AlarmManager.ELAPSED_REALTIME_WAKEUP,
  SystemClock.elapsedRealtime() + TimeUnit.MINUTES.toMillis(15)
  intent
);

This code won’t always work though. While it is guaranteed that the alarm will go off and the PendingIntent will be sent, because we used a _WAKEUP alarm type, the device is allowed to go back to sleep before the service is started.

It’s not explicitly documented, but both +Dianne Hackborn and +CommonsWare confirmed this. The workaround is to use PendingIntent.getBroadcast(), because it is guaranteed that the BroadcastReceiver.onReceive() will be always fully executed before the CPU goes to sleep. Inside that callback you have to acquire WakeLock start your IntentService and release the lock at the end of onHandleIntent() method.

This is where the WakefulBroadcastReceiver comes into play: its startWakefulService and completeWakefulIntent static methods encapsulate all the WakeLocks juggling, allowing you to focus on your business logic.

The first post in this series explained the first consequence on implementing the offline mode – performing the data migrations. In second part I showed a workaround for the rudimentary SQLite’s ALTER TABLE syntax. If you have checked the link to MigrationHelper class I mentioned, you migth have noticed that it’s just a tiny part of a larger library, which allows you to define database schemas. Note the plural “schemas”: the whole point of this library is defining both current schema and the schemas for the older versions of your app. This post explains why do you have to do this.

Let’s say in the first version you have the following data structure:

1
2
3
4
5
6

public static class User {
  public long id;
  public String firstName;
  public String lastName;
  public String email;
}

And the table definition for this table in your SQLiteOpenHelper looks like this:

1
2
3
4
5
6
7
8

private static final String CREATE_TABLE_USERS = "CREATE TABLE " +
    TABLE_USERS +
    " ( " +
    ID + " INTEGER PRIMARY KEY AUTOINCREMENT " + ", " +
    FIRST_NAME + " TEXT " + ", " +
    LAST_NAME + " TEXT " + ", " +
    EMAIL + " TEXT " +
    " ) ";

In the next version you decide to keep only the first name in a single field, so you change your data structure accordingly and perform the data migration. In the snippet below I used the MigrationHelper, but you might have as well performed the migration by hand:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

private static final String CREATE_TABLE_USERS = "CREATE TABLE " +
    TABLE_USERS +
    " ( " +
    ID + " INTEGER PRIMARY KEY AUTOINCREMENT " + ", " +
    NAME + " TEXT " + ", " +
    EMAIL + " TEXT " +
    " ) ";

@Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
  MigrationsHelper helper = new MigrationsHelper();
  if (oldVersion < 2) {
    helper.performMigrations(db,
        TableMigration.of(TABLE_USERS)
            .to(CREATE_TABLE_USERS)
            .withMapping(NAME, FIRST_NAME)
            .build()
    );
  }
}

Then you decide that the email field should be mandatory, so you change the schema and migrate the data again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

private static final String CREATE_TABLE_USERS = "CREATE TABLE " +
    TABLE_USERS +
    " ( " +
    ID + " INTEGER PRIMARY KEY AUTOINCREMENT " + ", " +
    NAME + " TEXT " + ", " +
    EMAIL + " TEXT NOT NULL" +
    " ) ";

@Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
  MigrationsHelper helper = new MigrationsHelper();
  if (oldVersion < 2) {
    helper.performMigrations(db,
        TableMigration.of(TABLE_USERS)
            .to(CREATE_TABLE_USERS)
            .withMapping(NAME, FIRST_NAME)
            .build()
    );
  }
  if (oldVersion < 3) {
    db.execSQL("DELETE FROM " + TABLE_USERS + " WHERE " + EMAIL + " IS NULL");
    helper.performMigrations(db,
        TableMigration.of(TABLE_USERS)
            .to(CREATE_TABLE_USERS)
            .build()
    );
  }
}

The code looks fine, but you have just broken migrations from v1 to v3. If there is an user with a null email field, the app will crash in line 13 above. But why, shouldn’t the email field in v2 schema be nullable? It should, but this migration uses the constant containing the latest schema definition with different column constraint.

The worst thing about this kind of bugs is that it might slip through your tests, because the crash happens only if you have a specific data before the application update.

You migth be tempted to define separate migrations from every old version to the latest one (in our case migrations from v1 to v3 and from v2 to v3) and always execute only single migration, but this workaround doesn’t scale. For each new migration you’d have to check and potentially update every existing migration. When you publish the app twice a month, this quickly becomes a huge problem.

The other solution is to make every migration completely independent from the others, and execute them sequentially. This way, when you define a new migration, you don’t have to worry about the previous ones. This means that when you upgrade from v1 to v3, you first upgrade from v1 to v2 and then from v2 to v2 and after the first step the database should be in the same state it were, when the v2 was the latest version. In other words, you have to keep an old database schemas.

As usual there are multiple ways to do this. You can copy the schema definition to another constant and append “ver#” suffix, but it means there will be a lot of duplicated code (although this code should never, ever change, so it’s not as bad as the regular case of copypaste). The other way is to keep the initial database state and all the schema updates. The issue here is that you don’t have a place in your code with current schema definition. The opposite solution is to keep the current schema and the list of downgrades. Sounds counterintuitive? Don’t worry, that’s because it is counterintuitive.

In android-schema-utils I’ve chosen the third approach, because in the long run it processes less data than the upgrades solution – in case of upgrade from vN-1 to vN it has to generate only 1 additional schema instead of N-1 schemas. I’m still not sure if the code wouldn’t be clearer had I went with duplicated schema definitions approach, but the current approach, once you get used to it, works fine. The schema and migrations for our example would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

private static final MigrationsHelper MIGRATIONS_HELPER = new MigrationsHelper();
private static final Schemas SCHEMAS = Schemas.Builder
    .currentSchema(3,
        new TableDefinition(TABLE_USERS,
            new AddColumn(ID, "INTEGER PRIMARY KEY AUTOINCREMENT"),
            new AddColumn(NAME, "TEXT"),
            new AddColumn(EMAIL, "TEXT NOT NULL")
        )
    )
    .upgradeTo(3,
        new SimpleMigration() {
          @Override
          public void apply(SQLiteDatabase db, Schema schema) {
            db.execSQL("DELETE FROM " + TABLE_USERS + " WHERE " + EMAIL + " IS NULL");
          }
        },
        auto()
    )
    .downgradeTo(2,
        new TableDowngrade(TABLE_USERS, new AddColumn(EMAIL, "TEXT"))
    )
    .upgradeTo(2,
        SimpleTableMigration.of(TABLE_USERS)
            .withMapping(NAME, FIRST_NAME)
            .using(MIGRATIONS_HELPER)
        )
    .downgradeTo(1,
        new TableDowngrade(TABLE_USERS,
            new AddColumn(FIRST_NAME, "TEXT"),
            new AddColumn(LAST_NAME, "TEXT"),
            new DropColumn(EMAIL)
        )
    )
    .build();

There are other benefits of keeping the old schemas in a more reasonable format than raw strings. Most of the schema migrations can be deducted from comparing subsequent schema versions, so you don’t have to do it yourself. For example in migration from v2 to v3 I didn’t have to specify that I want to migrate the Users table – the auto() migration automatically handles it. If the auto() is the only migration for a given upgrade, you can skip the whole upgradeTo() block. In our case that covered about 50% migrations, but YMMV.

If you go this way, your onUpgrade method, which usually is the most complex part of SQLiteOpenHelper, can be reduced to this:

1
2
3
4

@Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
  SCHEMAS.upgrade(oldVersion, mContext, db);
}

This part concludes the “offline mode” series. Here’s the short recap:

• If you don’t want to compromise on UX, your application should work regardless whether the user is connected to internet or not.
• In this case the user may end up in a situation when the application is upgraded, but not all data is synced with the server yet. You do not want to lose your users’ data. You’ll have to migrate them.
• If you migrate your data, you should keep the migrations separate from one another, because otherwise maintaining them becomes a nightmare.
• The best way to do this that I know of, is keeping the old schemas and always performing all migrations sequentially. To make things simpler, I recommend the android-schema-utils library.

In my previous post I mentioned a nasty SQLiteDatabase gotcha and recommended using the MigrationHelper utility I wrote. If you have checked this class’s sources, you might have noticed a weird code. Before getting the list of columns the table is renamed to the temporary name and then renamed back:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

final String tempTable = "tmp_" + tempTableIndex++;
db.execSQL("ALTER TABLE " + migration.tableName + " RENAME TO " + tempTable);
ImmutableSet<String> oldColumns = getColumns(db, tempTable);

db.execSQL(migration.createTableStatement);
final String tempNewTable = "tmp_" + tempTableIndex++;
db.execSQL("ALTER TABLE " + migration.tableName + " RENAME TO " + tempNewTable);
ImmutableSet<String> newColumns = getColumns(db, tempNewTable);

db.execSQL("ALTER TABLE " + tempNewTable + " RENAME TO " + migration.tableName);

private static ImmutableSet<String> getColumns(SQLiteDatabase db, String table) {
  Cursor cursor = db.query(table, null, null, null, null, null, null, "0");
  if (cursor != null) {
    try {
      return ImmutableSet.copyOf(cursor.getColumnNames());
    } finally {
      cursor.close();
    }
  }
  return ImmutableSet.of();
}

Initially the MigrationHelper’s code looked like this:

1
2
3
4
5
6

static final String TEMP_TABLE = "tmp";
db.execSQL("ALTER TABLE " + migration.tableName + " RENAME TO " + TEMP_TABLE);
ImmutableSet<String> oldColumns = getColumns(db, TEMP_TABLE);

db.execSQL(migration.createTableStatement);
ImmutableSet<String> newColumns = getColumns(db, migration.tableName);

It worked for a single migration, but didn’t work for multiple migrations – the helper method for getting the column set always returned the columns of first table. Since the query was always the same, I suspected the results are cached somewhere. To verify this hypothesis I added to the temporary table name an index incremented with every migration. It worked, but then I realized I need to do the same for getting the columns of the new schema – otherwise the helper wouldn’t work if the same table were migrated twice. This way the weird code was born.

But the same thing could happen outside of MigrationHelper operations, for example if you need to iterate through rows of the same table in two different migrations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

@Override
public void onUpgrade(final SQLiteDatabase db, int oldVersion, int newVersion) {
  if (oldVersion <= 1500) {
    Cursor c = db.query("some_table", /* null, null, null... */);
    // use Cursor c
  }

  // other migrations, including ones that change the some_table table's columns

  if (oldVersion <= 2900) {
    Cursor c = db.query("some_table", /* null, null, null... */);
    // try to use Cursor c and crash terribly
  }
}

So I checked the AOSP code for the suspected cache to see how the entries can be evicted or if the cache can be disabled. There are no methods for this, so you can’t do it with straightforward call, but maybe you can exploit the implementation details?

On ICS the cache is implemented as LruCache, so theoretically you could evict old entries by filling the cache with new ones, but there is one hiccup – you don’t know the cache size, so you’d always have to go with MAX_SQL_CACHE_SIZE.

Before ICS you couldn’t do even that – the implementation of this “cache” is just a fixed size buffer for SQLiteStatements. Once that buffer is full, no more statements are cached. This also has one more consequence – your app might work much slower on Android 2.x after upgrade from old version than after fresh install, because the db cache will be filled with queries used in migrations.

Fortunately the keys of this cache are raw SQL strings, so we can disable cache for migration queries by adding WHERE n==n clause with n incremented for every query (note that you musn’t pass n as a bound parameter – the whole point of adding this selection is to make the queries different and force SQLiteDatabase to compile another statement).

The question you should ask yourself is why do I have to know and care about all this. Isn’t SQLite smart enough to see that I’m trying to access the database using prepared statement compiled against old schema? It turns out the SQLite detects this issues and raises SQLITE_SCHEMA error (commented with “The database schema changed”), but Android’s SQLiteDatabase wrapper drops this error and happily uses the old, invalid statements. Bad Android.

I’m primarily an Android developer, so when I checked the Java 8 features list I thought there is a lot of cool stuff, by sadly I won’t be able to use them anytime soon. It’s the same case as AutoCloseable interface from Java 7. It’s available from API lvl 19, and seeing how long it takes Android community to unanimously drop the support for Froyo, Gingerbread and Honeycomb, I think won’t be able to use it before 2017. Anyways, good stuff is added to Java, but there is one cool feature from C# I do not see there: extension methods.

Let me explain to you what they are in case you haven’t wrote any C# code. In almost every code base there are simple utility methods which operate on a single object.

1
2
3
4
5
6
7

public static final class CollectionsUtils {
  public static <E> Collection<E> filter(Collection<E> unfiltered, Predicate<? super E> predicate) { /* ... */ };
  public static <F, T> Collection<T> transform(Collection<F> fromCollection, Function<? super F, T> function) { /* ... */ };
}

// usage
CollectionsUtils.filter(list, IS_NOT_NULL);

Things get ugly when you want to call multiple utility methods:

1

CollectionsUtils.transform(CollectionsUtils.filter(list, IS_NOT_NULL), TO_STRING);

C# allows you to add “this” modifier to the first parameter of static method, which basically tells the compiler to pretend that the objects of that type have a method with the same signature as our static method, sans the “this” parameter. Underneath it’s treated exactly as the ugly nested calls above, but it allows you to write the code this way:

1

list.filter(IS_NOT_NULL).transform(TO_STRING);

Syntactic sugar, but it goes a long way. I’ve intentionally choose the methods for this examples – whole LINQ-to-objects interface is based on extension methods.

Java 8 introduces a feature with similar name but completely different functionality: virtual extension methods. Simply put it allows merging the Foo interface and AbstractFoo abstract class with a reasonable implementation of some of Foo’s methods. For example if your interface has size() method you can add the isEmpty() virtual extension method with default implementation returning true when size() returns 0. So it’s a nice feature, but IMO less powerful than C# solution. Both solutions allow adding new methods with default implementation to interfaces you wrote without having to worry about backwards compatibility, but C# extension methods allows you also to extend 3rd party or even java.lang intefaces and classes to make their API cleaner or better suited to your particular problem.

I wonder why the C#-style extension methods weren’t added to Java 8. Maybe there are some implementation issues I do not see, maybe there is a conflict with another language features, maybe the powers that be think it would be inconsistent with the language philosophy. Do let me know if you have such information.

In first part of this series I showed that to implement offline mode in your Android app you have to implement data migrations. If you’re using SQLite database, it means you’ll have to use (or rather work around) it’s ALTER TABLE syntax:

So all you can do with it is adding the column or renaming the table, but in reality you probably need to alter a single column, remove column or change the table constraints. You can achieve this by doing the following operation:

1. Rename the table T with old schema to old_T.
2. Create the table T with new schema.
3. Use “INSERT INTO T (new_columns) SELECT old_columns FROM old_T” query to populate the table T with the data from the renamed table old_T.
4. Drop old_T.

Doing it manually is quite error prone though: for every migration you have to specify the new_columns and old_columns list. What’s worse, in 95% of cases you just want to list the columns common for old and new schema. Fortunately we can automate such trivial migrations by executing SELECT with LIMIT 0 (or PRAGMA TABLE_INFO) for both tables, getting the columns set using Cursor.getColumnNames(), and calculating these columns sets intersection.

You can write a nice wrapper for this yourself, but a) I already did it, so you don’t have to and b) there is a very nasty gotcha which would probably cost you few hours of teeth grinding, so do yourself a favor and check this repository out, especially the MigrationsHelper class. It automates the trivial migrations and allows you to define a mappings for situations when you rename the column or add a non-nullable column in new schema.

In the next two posts I’ll describe the gotcha I’ve mentioned in the previous paragraph and show some other non-obvious consequences of doing data migrations.