Mobile coding lessons i’ve learned from writing an Android application

I recently completed a major version of an Android client for work.  I guess no program is every really “complete”.  But, it’s at a good stopping point for the moment so i thought i would jot down some of the lessons i learned.  These can really apply to any type of development, not just mobile Android.

  • The network layer is unreliable
First of all, cell phones have flaky connections that come and go.  They might be on wi-fi, they might be on a cell connection. They might be in a tunnel.  Maybe they’re in airplane mode.  Doesn’t matter.  You can’t count on a network always being there, and even when it is there, you have to assume that it will be flaky and return strange errors.  It’s important to have a robust network layer that has retries built into it for various errors such as timeouts, unavailable, etc.  Don’t just assume the backend is down.  Assume first that their connection is temporarily out of sorts, retry a few times, and THEN give up.  That will eliminate 90% of the network errors a user has to be aware of.  Oh, and some phones, vendors, flavors of Android all have weird little network quirks that only they exhibit.  Yet another reason for a robust network abstraction.
  • Deal with threading issues

If you don’t want to die from a thousand little cuts, design your code to be thread-safe from the beginning.  Many phones now have multiple cores and really do execute things in parallel.  Add to that fact that any complex app will likely have multiple IO requests in flight at the same time (whether from the file system, database, or network) that can return asynchronously, and you are ripe for threading problems.  Deadlock, data switching out from underneath you, etc.  Use the threading tools. Synchronize, lock, atomic operations and especially the concurrent package are your friend.

  • Be asynchronous

Seriously.  Don’t block the UI thread for any reason.  If you’re loading a file from the file system, accessing the database, making a network call, or even doing some heavy number crunching, do it on a thread and call back to the UI when you’re done.  Android has some nice abstractions to help with this, including AsyncTask, and Handler classes.  You can also do traditional Runnable’s if you like.  But whatever method you use, make sure you have an async layer built into your app that’s easy to use.  Make sure your main application logic can deal with everything being asynchronous.  Your users will thank you for a snappy app.  Corollary: Don’t block the UI needlessly with modal dialogs unless you absolutely have to wait for a result.  Let the user do other things while your background processes are running.

  • Be aware of constrained resources

Ya this is a fun one.  You have very limited memory in which your application can run.  We’re not on a desktop here.  Some phones are far worse than others as far as how much memory they let you have.  Loading images is especially dangerous for running out of memory.  Have a strategy in place to only keep in memory what you need.  Don’t keep things lying around longer than you must.  Memory is fast, but disk is cheap and certainly faster than the network and definitely better to take a few milliseconds to reload an image from the drive all the time rather than running out of memory and crashing the program.  So load it from the network, save it to disk, cash in memory until you run out, then just have some type of LRU cache.

  • Understand and work with the application lifecycle

This one may be somewhat android specific, but the general idea is sound for anything:  Each activity (screen/window) has a lifecycle.  It’s created, setup, running, paused, and eventually destroyed.  It might also be resumed and restored in the middle of all that.  The framework provides rules about when all these things happen and how you should deal with them.  Make sure you do the correct thing at the correct time.  Expect that since this is a mobile environment with lots of stuff going on that your activity might be asked to pause or shutdown at any time, even mid-process.  Save off your state, be able to restore it, and know how to deal with data inconsistencies that might result.  Don’t expect that x then y then z will always happen.  X and Y might happen, and you’ve started Z, but then a phone call comes in and your activity goes away.  When you come back, what do you do?  Make sure you do the right thing.

Unit testing private java methods

Warning: dry tech article ahead.

As is often the case when beginning a new project, I like to get the ground rules set.  On a well done java project, unit testing is a given in my mind.  But one thing that’s always been a sticky point is how to unit test private methods and members.  In order to keep a clean class, you don’t want to change the scope of a method to public just so you can unit test it.  And you certainly wouldn’t want to put your unit tests inside your actual class.

Some other approaches include package scope or pass-through methods.  Again – this dirties your code just for the sake of a unit test.  After a few minutes pondering, i figured that reflection must be the answer.  And yes – there is a really nifty way to have your cake and eat it too.  Cleanly separate your unit tests from your code AND keep your private methods and members private.

But first, let’s hear the naysayers.  I was surprised to find that many people think you shouldn’t test private methods.  “If it’s private, it’s not part of the contract and the point of a unit test is to test the public contract of a class”.  I couldn’t disagree more.  Often times, business logic and important algorithms that only make sense to the class are encapsulated in private methods.  But in order for the “public contract” to work correctly, all of the underlying pieces must also work correctly.  Sure, you can indirectly test the private internals by assuming they work if you get the right output on a public call, but i just think it’s better to teach each of the individual cogs in the machine to make sure each piece does its job.  Then when you assemble it all together into a larger whole, everything just works.

Another counter argument is “if you test all these private methods, your test cases will be brittle”.  There’s a modicum of truth to that.  Chances are that if you stick to the public API’s, they will change far less frequently than an internal private method.  But i believe that if you properly design your code so that the private methods do one job and do it well, once you write your test, there isn’t much need to go back and ever mess with it again.  And if you do – be sure to update your unit test.

Ok, enough of the philosophy.  On to the technical details of how you actually accomplish this.  Let’s assume a class that has the following private member and method.

public class MyClass
{
    private int _myPrivateMemberVar;
    ...

    public MyClass() { ... }

    private String doSomethingInteresting(String s, List<Integer> li)
    { ... }
    ...
}

Now, how to access these from a unit test that’s not in the same class? Access it via reflection and change the access at runtime.

    //change the accessibility of the method and member
    MyClass c = new MyClass();
    Method method = MyClass.class.getDeclaredMethod(
        "doSomethingInteresting", String.class, List.class);
    method.setAccessible(true);
    Field field = MyClass.class.getDeclaredField("_myPrivateMemberVar");
    field.setAccessible(true);

    //call the method
    String myString = "foo";
    List<Integer> myList = new ArrayList<Integer>();
    String result = (String) method.invoke(c, myString, myList);

    //access the member
    int memberVal = (Integer) field.get(c);

And there you have it – accessing private methods and members at runtime from a unit test. Happy testing!

Garmin TOPO map

I’ve got a Garmin unit that i’ve had for a few years now (60CSx). We occasionally take it out to go geocaching or on trips, but we don’t use it nearly as much now as we once did. My phone and my wife’s phone both have GPS units built in, and there are some pretty nice programs to do navigation as well as geocaching right from them. And our car has a built-in gps unit with navigation as well. But sometimes it still comes in handy.

One thing that’s really nice is the topographical maps that can be loaded on it. The only problem is that if you load multiple maps (such as street maps and topo maps), only one map shows at a time, and it’s the street map by default. In order to show the topo map you have to hide the street map.

No problem, right? Except when i actually tried to figure out how to do this, it took about 15 minutes of google searches and pouring through pdf manuals to figure it out. Talk about non obvious! Anyway, here are the steps, for future reference:

  1. view the map screen
  2. press Menu
  3. select “Setup Map” and press enter
  4. select “info” and press Menu
  5. check the option you want (such as “hide street map”)
  6. enjoy your topo maps. Be sure to show street map later to get back on the road

RAID 101


The information for this post was taken from this article. The article discusses much more than just “what is raid”. It also details how to run software raid using Linux. As I just want to post about RAID 101, i’ve boiled down a few useful bits here. This is not going to tell you how to run RAID, only what it is.

RAID – Redundant Arrays of Inexpensivbe Disks

RAID is not a substitute for backups.

RAID is good for: the ability to combine several physical disks into one larger “virtual” device, performance improvements, and redundancy.

RAID is NOT good if: the RAID itself is lost in one way or the other (theft, flooding, earthquake, Martian invasion etc.)

N is used to denote the number of active disks in the array (not counting spare-disks).
S is the size of the smallest drive in the array.
P is used as the performance of one disk in the array, in MB/s.

RAID-0
The devices should (but need not) have the same size.
If one device is much larger than the other devices, that extra space is still utilized in the RAID device, but you will be accessing this larger disk alone, during writes in the high end of your RAID device. This of course hurts performance.
there is no redundancy
[why use? combine several smaller drives as one larger drive. useful when combined with raid-1]
RAID-1
has redundancy
can be used on two or more disks with zero or more spare-disks
maintains an exact mirror of the information on one disk on the other disk(s)
the disks must be of equal size. If one disk is larger than another, your RAID device will be the size of the smallest disk.
If up to N-1 disks are removed (or crashes), all data are still intact
reconstruction of the mirror will immediately begin on one of the spare disks, after detection of the drive fault
Write performance is often worse than on a single device, because identical copies of the data written must be sent to every disk in the array
RAID-5
can be used on three or more disks, with zero or more spare-disks
size will be (N-1)*S
parity information is distributed evenly among the participating drives (parity information can be used to reconstruct all data)
If one of the disks fail, all data are still intact
If spare disks are available, reconstruction will begin immediately after the device failure
Both read and write performance usually increase

There are no special requirements to the devices from which you build your RAID devices. you can build a RAID from a mix of IDE and SCSI devices, and you can even build a RAID from other RAID devices.

Python file upload

Not only am I using C++ at my new job, but i’m also doing some Perl and some Python. I actually have found that I rather like Python. It’s a great language for what we’re using it for. As an aside that has nothing to do with this blog post, this link shows the most popular programming languages. I am using 7 of the top 20 (as of July 2008) programming languages right now. :)

Anyway, back to the point of this post: I was very surprised to learn that Python 2.5x does not have support for uploading binary files over http (i.e. it can’t do file upload). What? That’s crazy. I happen to need this functionality for a task i’m working on. After some searching, i was able to pull together a bunch of pieces on the net and get a custom solution working. Many sites had pieces of the puzzle, but nobody had an entire working example all put together. Anyway, here is my solution:

import os, stat, mimetypes, httplib

def post_multipart(host, selector, fields, files):
    """
Post fields and files to an http host as multipart/form-data.
@param host: the hostname of the server to connect to.  For example: www.myserver.com
@param selector: where to go on the host.  For example: cgi-bin/myscript.pl or blog/upload, etc..
@param fields: a sequence of (name, value) elements for regular form fields.  For example:
    [("vals", "16,18,19"), ("foo", "bar")]
@param files: a sequence of (name, file) elements for data to be uploaded as files.  For example:
    [ ("mugshot", open("/images/me.jpg", "rb")) ]
@return: the server's response page.
    """

    content_type, body = _encode_multipart_formdata(fields, files)
    h = httplib.HTTPConnection(host)  
    headers = {
        'User-Agent': 'python_multipart_caller',
        'Content-Type': content_type
        }
    h.request('POST', selector, body, headers)
    res = h.getresponse()
    return res.read() 


def _encode_multipart_formdata(fields, files):
    """
@return: (content_type, body) ready for httplib.HTTP instance
    """
    
    BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
    CRLF = '\r\n'
    L = []
    for (key, value) in fields:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"' % key)
        L.append('')
        L.append(value)
    for (key, fd) in files:
        file_size = os.fstat(fd.fileno())[stat.ST_SIZE]
        filename = fd.name.split('/')[-1]
        contenttype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
        L.append('--%s' % BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
        L.append('Content-Type: %s' % contenttype)
        fd.seek(0)
        L.append('\r\n' + fd.read())
    L.append('--' + BOUNDARY + '--')
    L.append('')
    body = CRLF.join(L)
    content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
    return content_type, body

if __name__ == '__main__':
    post_multipart("my.server.com", "/cgi-bin/uploadphoto.pl", [("foo", "bar")], [("mugshot", open("/images/me.jpg", "rb"))])

Mac OS X Home/End key bindings

One thing that i absolutely cannot stand about the MAC is the non-standard behavior of the home and end keys. Every other OS i’ve ever used, when you press Home, it takes you to the beginning of the line, and when you press End, it takes you to the end of the line. If shift is pressed, it highlights to the beginning or end from your current location.

This page tells how to get that behavior in most OS X applications. The long and short of it is this:

Create (or edit) the following file:

~/Library/KeyBindings/DefaultKeyBinding.dict

Add the following to the file:

{
        /* Remap Home / End to be correct :-) */
        "\UF729"  = "moveToBeginningOfLine:";                   /* Home         */
        "\UF72B"  = "moveToEndOfLine:";                         /* End          */
        "$\UF729" = "moveToBeginningOfLineAndModifySelection:"; /* Shift + Home */
        "$\UF72B" = "moveToEndOfLineAndModifySelection:";       /* Shift + End  */
}

One notable application that does NOT respect the KeyBindings is Eclipse. Fortunately, Eclipse has its own mechanism for changing KeyBindings. Go to the preferences -> General -> Keys, and then change the Start, End, Select Line End, and Select Line Start mappings to be what you want.

Happy typing!

Checking in with AccuRev

At my new company we use AccuRev for source control. The license fees are insanely expensive and it’s extremely complicated to do such a simple thing as “check in some code”. Here is the process to do a check-in. AccuRev actions are in bold. (I’m documenting it here so i can remember it later on):

  1. Get the latest changes from upstream by doing an update. (Click the green lighting bolt – that’s the update button).
  2. If there are overlaps (i.e. conflicts that couldn’t be resolved automatically), do an overlap search, go through each file and manually merge (i prefer to use Eclipse’s merge tool here). When done, do a merge on each file. This will mark it as merged on your machine.
  3. Now that you’re up to date with the stream (i.e. branch from which you are working), do a modified search to see which files you’ve changed.
  4. For each modified file, do a Diff against Backed Version to see your changes and make sure you like them. Once satisfied which the changes, do a keep operation on each file to mark your local version of the file as being ready to send to the stream.
  5. Don’t forget to do a external search now so that you can see any new files that you’ve created that haven’t been added to the repository yet. For each file in this list, do an Add to AccuRev Depot operation so that the local files are ready to add to the stream.
  6. Now do a pending search to get a list of all files that are waiting to be promoted (i.e. moved from your local machine to the stream where others can get your changes).
  7. Select each file in the pending list and do a promote to actually commit it to the repository. Add your comment, and oh yes – you must have a bug # that you enter in order to add your code. Pretty much this is meaningless – everyone just makes a bogus bug # with some generic error like “scott’s code changes for iteration 2” and uses it over and over.

All done. That was easy, wasn’t it? HOLY CRAP!

My friend and co-worker Bruce has posted a blog entry talking about why we’re using AccuRev. Click Here for his review, which includes phrases such as “crazy expensive”, “arcane and fragile”, “steep learning curve”, “doesn’t make it easier”.

So why use it? It does do one thing very well, and that is merge changes effectively, accurately, and automatically when you have a huge number of engineers concurrently modifying large parts of the code base – thus avoiding “merge hell”. And that’s why we use it …


Update: I’ll add a few tips as i find them.

Tip #1: Another annoying ‘feature’ of AccuRev is that each workspace is associated with a hostname. And for some reason, the hostnames seem to change on me once in a while. I don’t profess to know why this is. But once it happens, you can no longer access your workspace to do updates and promotions. To restore your hostname, right-click on the workspace, click edit, then click finish. Walla – hostname updated back to your current hostname.

Windows XP SP3 breaks Windows Update

I just recently wiped one of my test machines clean and installed Windows XP (with SP2) and then installed SP3 to try it out before i install it on my other machines. Although the update went well, after I installed SP3 and then tried to install other Windows Updates (WU), they would all fail to install.

It turns out the problem lies with the Update Manager (UM). I found a blog post here that tells how to solve the problem. You basically must upgrade to the 3.0 version of the UM and then WU will work. Uhm, ya – but SP3 doesn’t bother to do that for you. Really smart Microsoft … I was about to reinstall without XP3 until i found this blog entry. Perhaps you should consider makes UM 3.0 part of your XP3 update process?

Here are the steps:

  1. Manually download the UM 3.0. It is located here:
    http://download.windowsupdate.com/WindowsUpdate/redist/standalone/7.0.6000.381/WindowsUpdateAgent30-x86.exe
  2. Open a command prompt and execute the following commands:
    1. net stop wuauserv
    2. "/[wherever you downloaded the file]/WindowsUpdateAgent30-x86.exe" /wuforce

      Don’t forget to include the ” marks if your download path has spaces.

    3. net start wuauserv

That’s it. You should now be upgraded to the latest version of the UM. In the comments to this post, someone said you don’t need to bother typing the net stop and net start commands. *shrug*. I guess if you don’t and it doesn’t work you can always try it again with them.

How to perform a Subversion (SVN) merge using the Subclipse Eclipse plugin

  1. Make sure all changes in your branch are checked into the SVN repository.
  2. Make sure you have the latest version of the trunk checked out from the SVN repository.
  3. Right-click on the trunk project and select team -> merge.
  4. In the from section, make sure the SVN trunk repository is selected.

    Example: svn://svn.myserver.com/reponame/prjgroup/project/trunk
  5. In the from section, select the revision radio button and make sure the value is what was current
    when the branch was made, or when the last time a merge was performed.

    Note: you’ll have to figure this out if you didn’t take a note when you created the branch!
  6. In the to section, uncheck the “use ‘from’ url” and make sure your branch is selected.

    Example: svn://svn.myserver.com/reponame/prjgroup/project/branches/mybranch
  7. In the to section, select the “head revision” radio button.
  8. Click the unified diff button.

    Save the file and open it.

    This will create a cvs-style “diff” file that will show you all of the changes.

    When you’re satisfied you have properly selected the correct root revision, click the merge button.

    This will perform the merge locally into your trunk codebase. It will NOT modify the
    SVN reposotiry.
  9. If there are conflicts, resolve them.
  10. Check in the trunk and the merge will be complete.

Feb. Geek Dinner

I guess better late than never, right?  Here are my notes from the “geek dinner” for February.

The topic of the evening was Agile Adaptive Management.

The first speaker was David Spawn – Westminster Business Professor.

What makes a good manager?

  1. can give constructive criticism; builds up confidence
  2. open to ideas; ability to listen (partner on the team rather than a hierarchical leader)
  3. integrity as a leader; keeps commitments

Some other audience-supplied values:

  • empowering
  • involved
  • deal with people as individuals (not cookie cutter)
  • communication; give and receive feedback
  • mentor
  • integrity
  • mutual trust
  • knowledgeable

Agile-Adaptive Management Model:

— Innovation –>
Purpose & Organizational Value
    Hire Great People
       Do Something Innovative
          Learn & Reflect
             Deliver Results
<– Knowledge —

Next we had Alistair Cockurn speak (http://Alistair.Cockburn.us) about agile development.

Why do we need agile?

  • solving a problem not yet understood and it’s changing constantly
  • creating a solution not yet understood
  • writing in languages not yet understood
  • to a compiler that doesn’t forgive errors
  • short on resources

everything you do is situational sensitive

a software project is a goal-directed finite cooperative game
    goal 1: deliver _this_ software (simple, finite)
    goal 2: setup for the next game (infinite)

goals 1 and 2 conflict because the games situation never repeats

What is agile?  Agile is not “pair programming”, “standup meetings”, “test cases” .. those are useful, preferred practices, but agile is adherence to the core values:

  • individuals & interactions
  • working software
  • customer collaboration
  • responding to change

Finally, and very important: agile has limits; don’t use it for a team of larger than 15 or a project of “life critical”.