Joe@Nitobi

January 12th, 2009

Recently, Nitobi got an Android Dev Phone, which is a copy of the T-Mobile G1 phone that Brock got earlier in the year. I’m currently using it as my primary phone while I figure out whether I should get an Android Dev Phone or the OpenMoko. I’m still leaning towards Android, but I have noticed that there are a few major usability and other bugs that I’ve found while being a casual user, and not a dev.

If the Android phones are going to beat the iPhone, they have to do everything the iPhone does. That means that they have to be able to sync with my computer. Now, the Google G1 is a dumb device as far as this is concerned, and it acts as an SD card reader. I originally thought that this was awesome, and copied my music to my 4GB MicroSD card that I put in the phone. However, I also decided to install another application, and BOOM. SD Card is Unreadable.

The Android Phone for some reason handles the mounting in software, and can’t deal with IO from the phone and copying files at the same time. If this is the case, it should lock the user out of the touch screen with a USB Connected screen, keeping functionality for an emergency phone only, BUT instead we can access most of the features. Only the RingDroid application is able to detect this operation. I’m most likely going to add this check to the Android version of PhoneGap, as soon as we get some serious I/O features. I was lucky and I was able to reformat the device with my digital camera. A lot of people won’t realize that it needs to be reformatted and will throw away bad storage. This can get fairly costly, since 4GB and 8 GB Micro SD cards aren’t disposable yet.

This seems to me like it’s usability fail. I’m going to load the new Cupcake Build on the device with the Multitouch patch on the Java Stack in a couple of weeks to see if the new build fixes the issue. I definitely consider this a major bug, and I’ll probably file this if it’s not resolved in trunk.

Other than this, (and the Battery issue that’s been blogged about EVERYWHERE), I’m pretty happy with the performance of the phone, and I’m hoping that Android can take off and be a challenger against Windows Mobile for all the phones that both techs and non-techs use.

*Update* - After I first posted, I found the bugtracker. I realize that I have to upgrade the device to see if this bug still exists. I predict it does.

Posted in Uncategorized | No Comments »

January 6th, 2009

Recently, for client projects and for our own Complete UI code, we have been looking more and more towards selenium testing for functional testing of the application. However, it’s interesting to see how different people approach testing, that it seems that people forget the differences between what a Unit Test is and what a Functional Test is.

A unit test is where you test a method for a desired result. This is where each method is tested to make sure that it produces the same expected result each time. When I did QA earlier, I heard developers tell me that the main purpose of Unit Testing was just to test banking software to make sure that it produced the proper result, and that because it wasn’t banking software, it didn’t matter. This isn’t true at all, and in fact Unit Testing should be done at the beginning of a project. This is why I am partial to rspec, since it keeps things clean, and allows me to properly think about what I am trying to do with my code before I go off and do it. This saves a lot of time at the end, where looks matter.

Then there is functional testing. Normally this is testing that doesn’t get done, or testing that would traditionally be done by your beta testers or some other form of unforgiving users who are not going to use your app because it does that stupid thing where it deletes the values of the fields and doesn’t notify them whether their data was accepted. This is what selenium is for. Selenium is software that fills this gap, since it allows you to test the interface before and after it’s styled and it forces you to think about making your site easy not only for you but for other people and robots to navigate as well.

The reason I mention robots is beacuse that’s what Selenium is. It will not be able to beat the captcha, and you will be stuck testing certain things yourself. However, if you wrote unit tests for your authentication or comment systems, you would know that it does work, and this wouldn’t necessarily be where you are looking to do functional testing.

When doing functional testing, the best thing to do is to try to take a black-box approach. The questions that you need to ask is this:

• What are you testing?
• Why are you testing this?
• Is this the most efficient way to test this?

What are you testing?

This is important. Knowing the goal of the test is part of the battle. If you have to make sure that a site is serving something that the browser will use that you don’t have access to in Selenium, like browser add-ons, then you need to re-assess how you are testing it. For example, in some cases, I write code in Python using only unittest, the Expat XML parser, and httplib so that I can make sure it’s
serving XML files correctly that would otherwise be used by the web browser (or Google Earth KML).

If we are using a browser that can read KML files, it should read the header and the header should have the right mime type. We don’t test for that here, but we do test to make sure that we are
returning something that can be parsed as XML instead of bad data that can’t be parsed. This perhaps isn’t the best example of this practice, but is a good idea.

Remember that you are testing the site, and that you are NOT testing the browser. It’s possible that the browser has a bug, but in that case, it’s probably a good idea to maintain a level of browser agnosticism, and that any browser-specific stuff should be tested using jsUnit.

Why are you testing this?

Occassionally there will be tests that make no sense. They may test nothing, or they may be so trivial that testing them may be pointless, and would inflate your test results. This could also be something that has been unit tested or has been tested already. Doing the same thing over and over and expecting a different result is insane, and it’s important that we keep a level of sanity. Therefore, repeating ones self when writing tests should be avoided and if you find that a step is being repeated in numerous tests, it’s probably a good idea to abstract that out into its own test and to move up that test’s priority since this test failing will cause all tests that depend on this step to fail as well.

Is there a better way to do this?

Selenium is supposed to be a substitute for point and click testing. However, it’s a robot that will only do what you tell it to. It should not know what is on the server side, and it should not be able to get past systems that verify whether you are a robot or not, such as Captchas. This is where manual testing should happen.

Writing tests isn’t as much fun as writing code. The more time that you spend writing tests, the less time you spend writing code. However, the more tests you write, the better your code tends to get. Here are some tips to ease the pain of selenium testing:

Try to use Firefox and Safari for testing as much as possible

You are testing a web application. Selenium can not tell whether a class looks right when it’s styled a certain way. It is not subjective, it is objective. The reason that you should use Firefox or Safari for the primary browser to be tested for functional tests is the fact that these browsers have getElementsByClassName. This is probably the most useful Javascript method for gathering elements to be checked. Have a collection of elements in a table and you need to verify that they all showed up? Getting the length of the collection that this method gets is the easiest way to do it. Need to make sure that certain fields are displayed or not displayed? Checking to see if this method isn’t nil will work. This will make life so much simpler for testing.

The lack of getElementsByClassName is Internet Explorer rather irritating, and you may want to add it yourself, either with an existing framework, or a homebrew hack. However as far as testing goes, this may cause bigger problems than you think, since your version of getElementsByClassName may contaminate other JS code and cause weird results that may not actually be errors but may be caused by you trying to be clever. Don’t do this! If the app uses a framework that does this, it’s fine, but otherwise try to avoid adding things that could lead to false positives.

Elements that need to be verified should have IDs or classes assigned to them

A lot of functional testing requires navigating the DOM. The less crawling through the DOM the test does, the faster the result can be arrived at, and the sooner we can get the tests done so that we can
get the actual code fixed. This also makes the code MUCH easier to be styled if the UI is deemed ugly or wrong. This also allows you to automate IE testing easier so it’s easier to find elements using xpath or by using the DOM.

Keep the tests short. One Test per Class:

Python is full of many different test runners of various quality. If you need to write a log of what was done, it’s a good idea to make sure that they are broken down this way, otherwise no tests will be run and you will have an empty file. Also, in python there are many different runners, such as XmlTestRunner, HTMLTestRunner and TextTestRunner that work this way. I recommend XmlTestRunner for Python because of it’s compatibility with CruiseControl. This makes it easier to have Java-based tests and Python based tests to be on the same machine.

Use Selenium IDE as a guideline only:

Selenium IDE is a great add-on to Firefox. The thing is that most of the time the Python/Java/Ruby/whatever that it produces is not suitable for anything past the most basic tests. This is painfully obvious if you have something like the Nitobi Grid where IDs are dynamically generated and are progammatically selected. If you rely on this without the ability to seriously customize the tests in Python or Java, you will hit a wall hard.

Have Patience:

Selenium tests tend to take over the computer you test it on as far as workspace. This is fine. I run my selenium tests in a VM so that I can do other things and let Selenium have free reign on my browsers and screen space. If not, and you’re testing a batch of processes, then I recommend walking off for a while and coming back. The reason they call this Selenium RC is because of the remote control functionality and the fact that this automation can eventually be done when you are not in the office, usually during a build, or at some point throughout the day.

Remember that it’s important to be rigorous when doing Functional Tests, and that it’s important to test real values. I highly recommend the ID and class advice since this will more likely not break your tests if minor design and styling gets changed. Of course, this is just advice, and this is not a magic bullet to fix your app. Take whichever advice you feel is the most applicable to your situation.

Posted in Uncategorized | No Comments »