10 Rules for a Successful Project

I've been part of many projects over the years. Some large, as in several hundred million Dollars, some smaller. Most of them has been less than successful. The latest project, however, is a resounding success. It is large, over 50 million Euro, it is relatively high tech, and it is hugely fun. It has also been an eye-opener.

Whole books have been written about project management, hundreds of them. I've read a few, but learned little. The books tend to be very general, and contains what in programming books are referred to as "patterns". I think they all have their value, but they need to be used with caution. You can not read these books and blindly apply your freshly acquired knowledge, you need to thoroughly analyze the applicability of each and every point made. (This applies to many or most books, obviously)

I think the books would be more useful if they simply contained a list of rules, which should take up a page or two, and the rest of the book should be used to elaborate on each point, and provide examples of how in practice the rule can be followed (it's a book after all, so it must by convention contain more than two pages).

So after very little deliberation, I hereby grant you unrestricted access to my accumulated knowledge of project management after a good couple of years in a well run project (and many years in less well run projects).

1. Understand the assignment
This is the single most important rule. When you were in school, I am sure the teacher before every exam told you to read the assigment carefully, and make sure you understand exactly what is required. This is fairly obvious, but I think you will find that too many projects crumble because of this alone. Make sure that not only YOU understand the requirements, but also all other members of the project team, as well as the customer. Many requirement-matrixes are blindly based on "hand-me-downs" (someone did something vaguely similar somewhere else, or someone stumbled across Google), so frequently the customer does not really know what a particular requirement means, and why it is there. This is just a fact of life, and it's better to recognize and deal with it.

If there are no requirement matrix, make one. Yes, really. Document whatever expectation the customer has before commencing the project. Then you have a baseline from which you can estimate cost and resources, and something to bargain with when the requirements change. Make the customer adopt it as his own.

Going through the requirement-matrix with the customer will ensure there's no unpleasant surprises when the project is done. Even if you've met all requirements on time and on budget, the project will still be a failure if the customer thinks he's getting something he's not.

In practice, we use an Excel workbook, with every single requirement on its own row (currently about 1200). We then go through each requirement, and try to "translate" it into something tangible (requrements are usually formulated roundly, so as to not impose restrictions on the actual solution). We then go over those "interpretations" with the customer, and ensure that we understand what the customer means by the requirement, and the customer "understands our understanding". Labour intensive for a while, but very, very useful. We also do a number of other things to those requirements, but thats the most important part. There are tools to assist in this work, but I think the mentality of ensuring you know what you are getting into is by far the most important. The tools will follow.

Almost as important as understanding the requirements is making sure the customer/PM understands your requirements. If you are to deliver a, b and c, you need x, y and z. Make sure any caveats are clearly documented. If, at a later date, these conditions are not met, you have documentation on whose responsibility this was.

As an example, I was once tasked with implementing an ERP-system in a startup GSM operator. Following my own advice, I tried to get a requirement matrix for the system. What is it supposed to do? What information do you actually want it to provide? There was no such matrix, so I arranged meetings with the stakeholders. Starting with the project owner. As it turned out, nobody really knew WHAT they wanted, and most of them didn't know WHY they should want it at all! You can imagine the success I would have had, delivering a system nobody knows what is supposed to do. I killed the project.

2. Well defined roles and responsibilities
Again something that should be self evident, but is sinned against time and time again. Every member of the team must know exactly what is expected of them, and what their authority is. It's hard to do your job if you don't know what your job is. I can not stress this enough. As I see it, it's the Project Manager's job to define and communicate these roles. If he does not do this, he is not a good PM! Simple as that.

Under the same heading I want to add "processes". Depending on the size of the project team, well defined processes are incredible important. They ensure people know what to do in certain circumstances, and also enforce #3 on this list.

3. Document it!
"It" referring to pretty much everything. If you are in a meeting, make sure minutes are taken and sent out. If it is a meeting where anything gets decided, you will have a written record of that decision. If it's a meeting where potential issues as discussed, you will have a written record of those troubles being alerted. If someone wants you to do something not previously agreed on, get it in writing. You will need it.

Making sure things are documented in this way not only serves to cover your ass, it also as a mental filter for those making the request. If they understand that things are documented and their request might come back and bite them on the behind, they will think more throroughly through their request. Which is a Good Thing.

4. Manage Change
Change will happen. There will be changes to your requirement-matrix, or schedule, or something else. Make sure these changes are documented, and that you clearly communicate any and all effects of that change. If you are in a project for a paying customer, having a clear requirement-matrix (as you of course will have after reading Rule 1), you can now recuperate any additional cost associated with implementing that change. If you do not have a requirement-matrix when the project starts, you are working blind. The cost estimates for the project was based on air, and it's hard to charge extra (or ask for more resources) for more air.

5. Interfaces
In any project you will have interfaces to other systems/parts of the organisation/something. It could be technical interfaces (protocol x/y/z), or it could be organizational (so-and-so are your responsibilities when operating this and that). Make sure those interfaces are documented.

Translation: make sure you have a document which clearly states what you will deliver, and what you need the other party to deliver, with a timeline. Make sure it is signed by both parties. Partially to cover your ass if the other part does not deliver on time, and partially to force the other part to recognize the interface.

6. Take responsibility - Establish trust in the group
If you are a Project Manager, or in any other position where you get to take decisions, take them. Remember, a bad decision is infinately better than no decision at all. And you need to take those decisions, even if you don't like nor care for them. That's the price you pay for playing the Big Man.

This is also paramount when it comes to establishing trust in the project group. If the people who work with you do not trust you or each other, they will never pull in the same direction as you. Which is a sure-fire way of cocking up a project. Projects are difficult enough when everybody is on the same page and pulling in the same direction, let alone having to deal with individual agendas.

If you are not in a position to make decision, you still need to earn the trust of your managers. Easy enough; deliver quality work, on time. Thats all there is to it.

7. Don't assume
Anything. Make sure "someone" reads through any and all documentation sent from, or used by, the project group. Any solution or process should be approved by a key member of the project team. If you have sub-contractors, make sure you have someone in the team with enough knowledge of the subject to ask the tricky questions, and understand if something fishy is going on. Yes, I have seen BIG projects fail because the project team assumed the sub-contractor had the best possible technical resources to do the job, and it was not really worth it to kick up a shit-storm to get resources on the project team who could do QA on the design-documents. (Turns out, there were no design-documents. The big, multinational tech-company was winging it. And failing.)

8. Get commitment
I've seen too many project staffed by 5% resources. "This guy will use 5% of his time, that guy will use 10%, together with all the others that will make up the total number of hours required". I've never seen it work.

I think it boils down to ownership: If I only have a marginal role in a project, I am unlikely to look really hard for potential problems, and take action to avoid it. Someone else must have done that already. But when ALL the resources are marginal, none will. I've turned down several projects because they only wanted a few percent of my time. It has nothing to do with money, I get paid the same either way, and all to do with success. I don't want to be part of a project I do not think will succeed. So, when you staff your project, make sure your key members are 100% allocated (or damn close). If you are a key member, insist that you are 100% allocated. If this is not possible, I would walk away.

9. Risk Management
Risk Management is one of those terms which are kinda hard to pin down, as it could be pretty much anything. I like to think of it this way; EVERYTHING we do in the project is rooted in risk management. Everything that is documented is documented in order to minimize the risk that we're not in control. Any product chosen is chosen in order to minimize the risk of it not doing whatever it needs to do properly. Any technical solution or routine is establised to minimize risks. The Excel-sheet in Rule #1 is used to minimize the risk that we miss our target. Having a concious view of risk management is important, as well as useful.

In practice, we do risk management sessions at various times in the project. For any big change, we assess the risks associated, and come up with ways to mitigate the risks. Then we make sure the risks identified is communicated to the "higher ups". Again, the mentality is the important thing. How you choose to implement it is up to you.

10. Test
After anything has been implemented (not only technical solutions), make sure you test it! Every testable requirement should be tested and documented, but also functionality that has not specifically been requested. I've never seen a requirement matrix that specifies that the users must be able to change the passwords on their PC, but it's a pretty big deal if it doesn't work..

This is also one of the things we go through when we discuss the requirements with the customer. At the end of the project, time comes to get paid. If it's a sizeable project, the customer wants to make sure you've delivered everything you've agreed to. Asking the question for every requirement "how should we document delivery of this requirement" is worthwhile. Once you can produce test-reports for the requirements, objections tend to go away. Which is another Good Thing.

There are more..
There's a lot more rules, but those are Top 10 in my book. I will mention one more, but without a number (or else it would be "Top 11 in my book", which doesn't really work..) A Plan. Really? A plan? Yes. A plan. Every project has a plan. But they are frequently badly formed, and not really followed. The plan should contain all your milestones, the most important tasks, and whatever interdependencies there are. That does not mean it should be as detailed as possible, but rather that the "right" tasks should be there. And it should be followed up. Every week the team should assemble and report the status of their tasks. If something is falling behind, the plan will show any snowball-effect, and solutions or new priorities should be implemented in the plan. The plan is not a static document, but rather a tool for managing the progress or lack thereof.

In general this last statement applies to many traditional tools in PM. They are not something you just need to have because the book says so, they are tools, and need to be used as such. If something really does not help with finishing the project, just drop it. Find something else that works.

A lot of what I have written can be interpreted as "Cover Your Ass" tactics. It's not. When everything is documented, uncertainty and risk is minimized. Everybody knows who is responsible. How often have you been in a situation where the opposite party says "Me? I thought you were supposed to do that?" Having it documented (and accessible) will aid in minimizing silly misunderstandings like that.

I do not suggest you let something slide because you know it's not your job. Quite the opposite, actually, make whoever is responsible aware that something is not up to snuff. This is perhaps the most important rule of all; have an environment in the project team, and include the customer in this, where everyone is looking out for each other! Making mistakes is not a hanging offence! (Making the same mistake twice is, however)

An idea for a new Storage Engine for MySQL, Part II

So, a few days ago I blogged about an idea I had for a new storage engine for MySQL. The MySQL API is C++. so it would require me to actually learn C++. Which I tried, but it hurt my eyes. So I decided to just go ahead and write a prototype in C#, just to quickly test out the basic premise; reducing/eliminating storing multiple copies of the same data, using B-trees and hash-values. (And gaining performance in the process..)

Since then I've spent 4 days in Istanbul, which was brilliant, but before I went I made a quick prototype, using .Net 4.0beta1. The preliminary results were encouraging enough to motivate me to polish it up a bit. In essence, I took 70MB of logs, and imported it into the storage enginge in 18 seconds, resulting in about 22MB on disk. (That is 32bit-aligned, mind you, so if size was the only concern it could be trimmed down quite a bit) Fairly simplistic, and for now everything is stored as byte-arrays, it doesn't even support deleting data. But with a little brain-work, I guess it could be a useful way of quick and simple permanent storage on .Net. (I really don't like the idea of requiring SQL Server for trivial stuff)

Not the first project with that aim, I know, but it is an interesting learning experience nonetheless. Incidentally, memory mapped files are supported in .Net 4.0, which is cool. In time, I hope to also make a MySQL-version, which would make it possible to use the same datafiles from a .net-application and MySQL, which might be vaguely useful; for instance writing a small .Net snippet to create a database from some moderately complex dataset, and exporting it to MySQL with a simple file-copy. Which I would file under C, for "Cool".

So, this is where I am at the moment:

1. I've got a crude XML-format for defining a row-format, and a parser for it.
2. A class-hierarchy that works fairly well
3. A b-tree implementation that seems to work (I'm sure theres a few off-by-one's, but that's hardly the point)
4. A data loader which loads in data based on the supplied dataformat
5. A cmd-line app that lets me play with it

I can load data, and retrieve it as arrays of strings based on row-numbers. Each column uses two files, both memory mapped. One datafile contains an entry for each row, which is simply the record-number in the b-tree where the data can be found. The B-tree file contains entries for nodes and data, stored in pairs. A node-entry, followed by a datablock of static size.

A variation of this is for really small fields, for instance int's, in which case the column simply contains the values directly in the row-entry, instead of a pointer to the b-tree. These columns does not have b-trees.

I didn't bring a laptop to Istanbul, but I did think about it every now and then. So here is the current TO-DO list:

1. Make stress-tests. Randomly retrieve lots of rows, timing it
2. Add more statistics, maximal search length, number of levels in the b-tree, timing searches etc
3. Single datafile:
 - Extents. Need map of extents, each column has one or two extents * n, data and row
 - Need logic to get new extend when one is full
 - rowCount on table-object, not column
 - Extents handled by table-object
 - mmap one extent at a time
4. Support deletes
 - Use free-lists
5. Support variable datasizes
 - Another segment for actual data, for a total of two or three:
 - Non-optimizing:
  - 1 segment for rowdata, with pointers to another segment with valuedata.
  - a single list for fixes sizes 
 - Optimizing:
  - 1 segment for rowdata, 1 with b-tree, another for value-data
  - the b-tree requires two reads anyway, so should not take longer (ValEntry+Data)
6. Create general library for .Net
 - Need decent way of retrieving data, apart from row-number
 - GetRow on Table-object, return row as object[]
 - RegEx'es on dataload
7. Create MySQL StorageEnginge using same dataformat
8. Indexing?
9. Store data as native datatypes, at least int and char[]
10. Profile it, optimize
11. Verify that 32bit-aligne actually improves performance.

An idea for a new storage engine for MySQL

This week I've spent some time analysing about a gigabyte of logs from an old IIS5.0 server. The logs are in the standard text-format, so I quickly decided to dump it all into MySQL.

Using MySQL for this sort of thing (or any database, for that matter) works like a charm. Yes, of course I could have used Perl, or for this particular case, any number of loganalysers, but I find that working with SQL is simply quicker and more convenient in practical terms, for on-the-fly one-time analysis where I don't really know up front what I'm looking for. It also allows me to use MySQL to aggregate the data, and use Pivot-tables in Excel to chew through and graph it. It simply works well for me. (And no, I don't care that other open source tools exists. I want it to work now, not after fiddling around with it for hours. And Excel is already installed on my laptop.)

Ideally I'd like to just dump the whole thing in Excel, but Excel will only allow you some 65000 rows, which is not nearly sufficient. My dataset is about 6.4 million rows.

Anyway, importing the logfiles was quite easy, a small commandline to clean the data (think grep and sed), and load data infile... from mysql did the trick. At a leisurely pace to be sure, but it did it. (I have MySQL running on Linux in a virtual machine, so I'm not blaming this lack of speed on Linux nor MySQL..)

So, while I was working with this data it struck me how much redundancy there was in the columns. For instance, in 6.4 million rows, I have only 2479 unique IP's. This is a quick-and-dirty thing, so I didn't spend time optimizing anything. The client IP-address in my table is varchar(15), because xxx.xxx.xxx.xxx is 15 bytes. The log-files are dumped as text, keeping everything as quick and painless as possible. In essence, the database is storing 6.4 million ~15-byte values, for only 2479 unique values. Or, to state it in real numbers, my database has 70,095,785 bytes of raw data stored in the IP-address field, not counting any overhead. The unique values sums up to 30,228 bytes.

In a production environment, you might create a second table with uniqe IP-addresses, and only reference that table from the "log" table. You'd save on diskspace, but your SQL would be more cumbersome to write. As I said, this was ad-hoc, and I don't need any extra headache on this type of work.

I've looked at the numbers, and the pattern seem to be repeating. The data is about two months worth, so the date-field only has 60 unique dates. There are, however, 6.4 million dates stored in the database. The request URL has only 657 unique values, yet I am storing those 6.4 millon times. It all adds up!

I've looked around for an alternative storage engine, but without much luck. Packed MyISAM-tables compress by rows, which won't do me any good. The Archive-engine the same, if I read it correctly.

Here I think it's time to pause and explain that I am not a DBA, I am not a programmer by trade or education (although I've have programmed in several languages, for my pet projects), and I do not intimately know MySQL. I use it occasionally. I browsed the source for the first time yesterday, and read a couple of Internets about the storage engine API. So the rest of the post should be regarded with scepticism by those looking to learn, and mercy by those who already know.

With that in mind, I'll take a couple of minutes to try to explain what the Storage Engine is, and why I'd like a new one. An RDBMS (or Relational DataBase Management System, MySQL is one) consists of a lot of pieces. Some part parses SQL, generates executionplans, optimize, and all sorts of more or less interesting bits, all of which are important. The Storage Engine is the bit that actually puts your data onto your disk (or wherever you want it). In MySQL, this part can be changed, depending on your needs. For instance, you have a storage engine that never stores anything to disk, it only keeps it in memory. This would be convenient for things that does not need to live long, but needs to be fast. The rest of the RDBMS doesn't know much about this, it only knows how to ask the storage engine for data. It really doesn't know what the storage engine does with the data, or how. It just wants to get what it asks for.

Likewise, when you write a select-statement, you don't need to worry much about which storage engine is used. It just all magically work. So you can use different storage engines, with different properties, just as long as those storage engines follow the rules, i.e. the API.

For a better explanation, check out the official documentation.

So back to my plan. Here is the crux of that I want: A storage engine that automatically checks if any value stored to a column, has been stored before. If so, simply point to the existing datablock. (And use Copy-on-Write for updates). In this particular case it would save me a LOT of data. I haven't crunched the numbers properly yet, but keep in mind that the entire Gigabyte of logfiles compresses to about 17MB, so there's a lot of redundancy in there.. Potentially I might even be able to hold the whole table in memory at once, which would speed up things quite a bit.

Think of it like this: a list for each column in a table, with one entry for every row. Let's call this the "row-list". Another list for each column in a table, with one entry for every value. Let's call this the "value-list". Each entry in the row-list is just a pointer to the correct entry in the value-list.

So, when an Insert comes along, the engine would look at each value, compute a hash-value, check this value against the value-list (or B-tree) for the relevant column looking for a match. If found, insert the pointer to the corresponding node in the value-list into the row-list, and increase the reference-counter on the value-list node. If not found, create a new node in the value-list, and point to it in the row-list.

A Delete would simply be decreasing the reference-count on the value-list, deleting the node if we reached zero, and delete the node in the row-list.

This all sounds frightfully involved, but my gut tells me it would pay off by reducing disk IO and memory-footprint. (Not sure if cache-locality might become an issue, though)

A quick estimate of datavolumes goes like this: I'm guessing I'd need 4 bytes (a 32-bit pointer) per row per column, in the row-list. Then I'd need a 32-bit hash-value and a reference counter on the value-list, plus the value itself. Total 4 bytes per row, 8 bytes per unique value, plus the actal data. I'm comparing against the raw data, not making any assumptions on MyISAM overhead. (I would also probably need some other bookkeeping datastructures/fields, but I guess MyISAM also uses those, so it evens out. In my head it does.)

A quick graph in Excel tells me that on a 15 byte value, I'd save on data up to about 48% unique values. In my dataset the unique values are less than 1%, so there is potential for BIG savings, as demonstrated with the IP-address example. I estimate 32-bit pointers are more than sufficient, remember we're talking about unique values. If you have more that 4 billion of those, you should probably look at a different storage engine! (The corresponding number for 64-bit pointers, incidentally, would be ~22%..)

Of course, the storage engine would need to be smart enough not to use this scheme for small fields. No sense in replacing a single byte-value with a 12 byte overhead..

One other interesting aspect is that I'd have 'free' indices; if value A matches an expression, this would apply to ALL rows that points to that node in the value-list. So there would be no need to evaluate those. I don't know if I would be able to get the optimizer to utilize this in any meaningful way, though.

I think this storage engine would be very useful for things like logs that has a lot of redundancy in it. It would, as far as I understand, cut down the importance of getting your data to normal form drastically, and might even throw in some nice enhancements for 'free'. The write-speed would probably be lower than, say, MyISAM, but that would be fine for the sort of scenario I'm looking at. It may on the other han turn out to be higher, for certain workloads, since there'd be way less disk-IO.. For both read and write. But that's speculation/whishful thinking.

So, step one in my plan for World Dominance: Learn C++.

Videogames, part II

So.. Number 2 on my list of annoyances in videogames (in no particular order). Superfluous Press whichever Button! This actually stopped me from playing through the greatest videogame in the history of man kind (in fact, the history of the universe, judging by it's diciples); Metal Gear Solid 2. The number of keypresses required just to be able to move an arthritis-ridden, stooping, smoking old geezer around the screen proved too much for me. I did not even finish the first level. It went straight to Gamestop, where I could at least recuperate some of my money. So much the better that the gamestudios never see a dime of that extra value generated from the second hand market. It is my one and only way of letting them know that I do not appreciate them wasting my time. By the way, MGS2 is by no means the only one, it just happends to be the worst one I've experienced so far. The post isn't about MGS2 in particular, but about consolegames in general.

First, there's just about 600.000 logos, pictures og god knows what, god knows why, and videos. This all invariably results in vigorous keypresses, on just about any key there is on the controller. I've also resorted to picking up any other controllers connected to the console in question, just in case there's a bug. To no avail, ofcourse. Who, in their right mind, thinks that I care who made the soundengine?? Or, more accurately, who thinks that I care about it so much, that I will find this information interesting, not only before I play the game for the first time, but every single time I play it?? This can not wait untill the credits roll?? This, I think, can be likened to prominently, for 15 minutes, dispaying the name of the gaffer before every movie shown. (No, I don't know what a 'gaffer' is either. But I know every movie have one.)

Then there's the "Press Start". Ok... Well, I've spent some time considering this, and I've come to the conclusion that it might be excusable, in the event that the game console has more than one controller attached. If not, NOOOOOOOOO! Get it? Just no. We, as a species, have placed people on the moon. The Cell-processor is, as far as I'm concerned, the grestest feat since the Alpha-processor, with a straight lineage to the wheel. It's a marvel. A testimony to human ingenuity. Yet, there it is, all nicely tucked into a sleek black console in my "study". (That's what my wife calls the room where she puts all of my stuff.) So don't tell me we can't make a videogame that doesn't require me to tell it that I want to play. It's on. That means I want to play it.

Well, so I've accepted the Press Start.. For now. So now what? Well, now comes, if you're lucky, the "Please select a storage device". Hm. I only have the one, so I guess I'll choose that one. What would any reasonable intelligent being guess, if pushed? There you go. Gamestudios are entirely staffed by an alien race, come to this planet in order to subdue us all into a mindless dance of pressing buttons, for no apparent reason. They, meanwhile, will probably ravage the women we ignore while in front of our consoles. Meh.. The game, by the way, needs to know which storage device to use every time it is started..

"Please select storage device" is only acceptable if there is more than one storage device! And only, only the first time the game is started. How hard can this be??

Well, that's three (or many). Time to play! Nope. First, they feel the need to inform you that this particular game has a cutting edge feature. One feature so new, so inventive, so revolutionary, that reminding you of it every time you start the game is entirely justified. Nay, required, even. Nevermind that you've logged in the better part of your toddlers life playing the game, nevermind that you've started the game far more times than you've called your mother in your entire life. No, the task they perform, and what you need to acknowledge with a keypress, is that of saving your progress. And it is of vital importance that you do not, I repeat not, under any circumstance, within the timespan of 2 ms take a dive towards your gameconsole and yank the out powercable. Good to know.

Then theres the Main Menu. Dear Lord.. I don't even know where to begin with that one. I've started games on my console literally thousands of times. I do not own thousands of games. That leads to the inevitable conclusion, again if you are a reasonable human being, that I continue where I left off far more times than I start a new game. In short, the number of keypresses from I start the console with a gamedisk in it untill I'm exploding slugs with a flamethrower should be Nil. If you insist, I can settle for one. "Do you want to continue where you left off [Yes][No]". That would be sensible. That would make me happy.

There's more keypresses. Sooo many more. But I think I've made my point. Superflous keypresses are bad.

Videogames, and how they annoy me!

Given the previous topics on the blog, it should come as no surprise that I own videogame consoles. Several, in fact. Xbox360, a PS3, and even a PSP. I don't have any games for that one, I simply like the fact that it will, if I should feel so inclined, allow me connect it to the PS3 and play wichever game is running on the PS3, on the PSP. I can not for the life of me see how that could ever be even remotely practial, but I do like the geek-factor. Yes, I suppose I am a geek.

What is perhaps a bit more surpising, given my apparent interests, is that I am married. To a woman. Who's not a geek. I am married to a real, live, non-geeky woman, who cares about non-geeky stuff. Such as what colour things are. If I should phone home one day with news of a recent purchase of, say, a car, her first response would be "oh, god..". This would be muttered. I would not, at this time, offer up more information, as "oh god" is not a question, and the muttering does not invite to share my excitement about said purchase. So there would, very likely, be Awkward Silence. The very next response would in all likelyhood be "what colour is it?"

I think this is a key difference between men and women, the properties of objects which they care about. (This, and the ability to laugh at Monty Python. Ofcourse, I am stereotyping here, but I am allowed to. It's my blog, and the Internets are well suited for this sort of thing. ) Now, speaking for myself, I would not inquire about the colour of anything unless this information would be of importance in actually locating this thing. I would, for instance, ask about the colour of my wifes new car if I had never seen it before, and was looking for it. Me, and most men I know, would rather inquire about things like horsepower, the number of doors, engine size and fuel, properties which concern the utility of whichever object is in question. Women seem to favour estethics. Nothing wrong with that, I like things that look good, but I can't help feeling that any object you bring into your existence should serve some purpose.

This also extends to the topic of the post, videogames. I don't know where to begin.. Most videogames annoy me to an extent hardly experienced by mankind before. The level of annoyance in Stalin when Hitler crossed the border into mother Russia, for instance, would be relatively minor in comparison. I quite often ask myself what these people are thinking of. So I think I shall start a list, right here, of Stupid Stuff to do in videogames. Lets start with cutscenes.

Cutscenes, for the uninitiated, are the short movieclips between gameplay. Some times they are there to convey information of importance to the gameplay, but most of the time it's there, as far as I can work out, for two reasons: 1) To show off how good they are at making things look exciting, and (far worse) 2) to fill up the time with nonsens so they can say "20 hours of gameplay", when there is in fact 2 hours of gameplay and 18 hours of mindless drivel.

I have in my flat the following devices which will let me watch moving pictures: 4 laptops. 1 PS3. 1 Xbox360. 1 PSP. 2 TV's. 1 DVDplayer. 1 PVR. Any one of these devices will show me movies and TV 24/7. So why, my dear game-studio exec, do you think I purchase a $60 game? To watch movies? The answer, as the more perceptive of you may suspect, is no.

Are all gamestudio exec's failed Hollywood directors? Are they like the security-guards on the subway, who failed to get in to police academy or the army, and now has to settle for some low-end job? Are they constantly thinking about that fantastic movie-deal they'll get, just as soon as they're discovered, by someone who buys videogames to watch movies?

Cut-scenes annoy me. The only thing that annoys me more than cut-scenes, are cut-scenes that can not be skipped! Yes, they do this. They make you watch stuff you do not really care about. There is no fast-forward, because they've gone to the trouble of making this, and you're going to watch it, dammit!

There really is no excuse for "non-skippable cutscenes". I am playing a game. You may think the game is so subtle that there's no WAY anyone could play it without the crucial information conveyed in the cutscenes. You'd be wrong. 99% of the time, it's like this: if it moves, shoot it. End of story. If there really is important information to be conveyed, there are other ways to accomplish that besides forcing me to watch it played out on screen. Text springs to mind. Text has worked for millennia. I predict a bright future for it.

Bootloading the Arduino over XBee ZB, Part II

Right, so I got the Arduino to load over XBee with the default bootloader. But I am still stuck with having to manually hit the reset-button on the Arduino.

So here is my plan. (I've seen various variations on this on the 'net, but not exactly what I want. Someone has probably done it, before, though. I don't care.) I wonder if that whole CTS hardwired to Reset can work for us his time. What I want is for the DTR, when received over the USB-connection by the XBee Coordinator, to magically reset the Arduino. This should in theory be very simple with the XBee shield.

• On the XBee EndDevice, set DIO7 to mode 4, which is "Digital Out, Low". (ATD7=4, if you're doing it by hand)
• Then, on the XBee Coordinator, set DIO7=Input (ATD7=3), and IC=80 (IC is a bitmap of which inputs to monitor, 40h is a bitmap with only the 8th bit set, counting from IO0)
• Finally, solder a jumper on your XBee explorer, tying DTR and CTS together. (Actually, this should probably be a resistor.. I'm guessing CTS is an input on the FTDI. And Maybe it should be possible to revert this somehow, I think X-CTU depends on CTS and DTR for firmware-loading. Minor details!)
• Done!

Looks good, no? This should work because the XBee's are quite intelligent little things. Because we set DIO7 on the Coordinator to Input, the XBee should automagically send this information to whichever XBee is addressed in the Coordinators DH/DL-registers (you did program those, didn't you?). And since we hardwired DIO7 to DTR, whenever AVRdude asserts DTR, the remote XBee will assert DIO7. And, as we know from bitter experience, DIO7 is hardwired to Reset on the XBee shield.

Sounds simple, doesn't it. Perhaps too simple.. I've struggled with this for a few hours, but no matter what I do, no matter what config, no matter what firmware, I can not get the target XBee to change its outputs.

I've concluded that this functionality ("transporting" Digital IO-lines) only works on Series 1 XBee's. Something that is not clear in the documentation. (As a sidebar, I would like to note that the XBee documentation in general is not all that great..)

Anyway, I've opened a case with Digi on this, hopefully they will come back with a clear answer. I'll update if they do!

Update! I've heard back from a very pleasant support-person at Digi, and he confirmed my suspicion. This is not supported on the XBee ZB, and it probably never will be. They also confirmed that this does work on their series 1 radios, the XBee 802.15.4.

So, what would the workaround be?? Well, I've verified that asserting DIO7 will reset the arduino. So I guess I'll write a small wrapper for AVRdude, which sends a package to the remote XBee instructing it to set DIO7 high, then low, and then fires up AVRdude. Might work. The bad news is that it will require the API firmware, at least on the Coordinator. Don't yet know if API Coordinator with AT EndDevice will have the desired effect. I do not yet know if I can make this work, if the EndDevice needs to have the API firmware in order to support remote commands, I don't think it will send the data straight out on the serial, but rather in a frame-structure. Meaning I'd need a custom bootloader in order to decode the data.. Hm.

So I suppose there will be a "Bootloading the Arduino over XBee, Part III". Sometime.

Bootloading the Arduino over XBee ZB

I was playing with the XBee and Arduino again, and reading a lot of posts about how to bootload the Arduino wirelessly over the XBee. It was a fairly intricate process, involving among other things a custom bootloader. Now, it should be said that most of these posts were fairly dated, so there may be something that's changed. Because when I tried it, it worked on first try, with the default bootloader.. Mysteriously.

Setup: XBee Coordinator on XBee USB Explorer on COM5, XBee EndDevice on Arduino Mega.

(BTW, if you read my previous post, this will not work with the rerouted Serial. Put both jumpers to 'XBEE' on the XBee Shield)

Anyway, here's how I did it.

• Set up both XBee's to use 57600 baud.
• Upload .hex-file using AVRdude. Here is my cmdline:

 avrdude -Uflash:w:ledXBee.cpp.hex:i -cstk500v1 -P \\.\com5 -pm1280 -b 57600 -D

• At the precise moment you hit Enter, also press the reset-button on the Arduino. You may have to try a couple of times. Basically, methinks its because DTR isn't transmitted to the Arduino wirelessly, so you have to perform that duty yourself. I have an idea on how to fix it, but it requires some soldering..
• Done.

A screenshot may be in order:

Ok, so that works like a charm. I must explain that the XBee's are 4 inches apart, and the binary is less than 3K. I don't know if it'll scale in size or distance.

So, over to that last pesky bit, having to hit reset manually.. But that will have to wait to another day.

XBee woes..

As I mentioned in my last post, I also procured a couple of XBee's and the XBee shield. I don't exactly know what the big idea with that shield is, as it really is a pain. In three ways:

1. The CTS pin is hardwired to the reset-pin of the Arduino. You'd think that would be prominently displayed, but it isn't.
2. It uses the same Serial as the USB. I have the Mega, so it might have more serials than the other Arduino's, I havent checked. (That would be the only reason I can see for doing this)
3. It only connects a few of the pins.

Before I go on, I think I should explain my setup here, as it may to be crystal clear from the text alone..

1. Arduino + XBee Shield + XBee (EndPoint), connected to COM4
2. XBee Explorer + XBee (Coordinator), connected to COM5
3. (Note to self: use bigger fonts)

So. Annoyance #1, the CTS-pin. That one threw me for a loop for a few hours. I have the XBee EndDevice on the Arduino (XBee ZB EndDevice, firmware 2x64). The Xbee ZB Coordinator is connected to my laptop, on the XBee Explorer. I uploaded the example sketch from Arduino "PhysicalPixel", which simply turns the LED on pin 13 off and on with the reception of a H or L on the serial-port.

With the default AT-firmware, I can now open the com-port where the XBee Coordinator is, set the DH/DL registers to the address of my EndDevice, and send H's and L's, observing the miracle that is a wirelessly transmittet blinking LED. Big Whoop.

This was, in itself, quite the little victory for me. Only, when the LED was on, I would ofcourse sit back and marvel at this little light (harbouring thoughts which included the words "evil" and "genious"). Only to observe that it turns itself off! For no reason what so ever! The code is so simple that it can not be a bug. So what is going on??

This is when Annoyance #2 hit me. Since the XBee and the USB FTDI shares the same serial-port, you must choose which one wins. This is done with a couple of jumpers, but it means that you can't simply add a few lines of code to the arduino to make it phone home with what is going on. Which makes it annoying to debug..

The "cure", so to speak, is exceedingly simple, at least on the Arduino Mega. Pull off the jumpers, connect the middle pin on each of the two jumpers to pin 18 and 19 on the arduino. It will then use Serial1 (as denoted in the Arduino GUI), instead of Serial, which allows you to fill your code up with "Serial.write("Hello");", and understand what is going on. (I simply use two instances X-CTU, on each of COM4 and COM5.)

(Note that in the picture I've left the jumpers connected to one pin, so I don't loose them. The blue socket is connected to the middle of the three pins, although it doesn't look that way in the photo..)

Very helpful, that.

Anyway, turns out the solution for Annoyance #1, Mysteriously Resetting Arduino is also extremely simple. When using X-CTU to configure your EndDevice, simply set IOD7=0 (or use ATD7=0, same thing). The XBee will no longer reset your Arduino simply because it wants to sleep.

Now, Annoyance #3. This one really occured to me while playing with this. Finally solving the two previous issues, I made a small program to let me connect to the EndDevice XBee on the Arduino through the USB-port. Very simple, read a byte from Serial, send it to XBee on Serial1. This way I could verify that the Arduino got the transmissions from the XBee Coordinator (by watching the Magic LED), and I could also enter command mode on the XBee EndDevice without pulling it off the XBee Shield.

This in itself was easy enough, but I noticed that the Xbee EndDevice, when sleeping, would not wake up from data on the serial-line (i.e. from the AVR). I don't know why. When I first sent something on the wireless side, it would wake up. As long as its awake, "talking" to it through the Arduino works just fine, but it will eventually go to sleep.

So. Another small "adustment" to the Xbee shield. First of all, I enabled Sleep Mode 5 (ATSM5) on the EndDevice. This sleepmode allows the microcontroller to wake the Xbee through pulling pin 9 (SleepRQ) on the XBee low. Fancy. Only, Pin 9 isn't connected anywhere on the Arduino! Annoying. Also, the XBee is 3.3v, while the Arduino Mega is 5v.

I "solved" this by soldering a 1.4Kohm resistor between pin 9 on the Xbee-socket and Pin 7 on the Arduino. Now, when I send anything on Serial (i.e. USB connected to my laptop), it pulls pin 7 (connected to pin 9 on XBee), low. This causes the XBee to wake up, and it is under my command. World domination must start somewhere, and I now consider myself well on the way.

A picture for the curious:

In case you're wondering why I chose a 1.4Kohm resistor, here's why:

• The Arduino is 5v, the XBee is 3.3v
• The XBee can sink maximum 4mA
• The Internet told me 5V through 1.4Kohm equals 3.57mA. So won't fry the XBee.
  

So, in conclusion.. I now have a setup which allows me to

1. Send random crap from my laptop, through the XBee Coordinator in COM4, to the Arduino.
2. The Xbee EndDevice does not randomly reset my Arduino
3. The Arduino can phone home over COM5 to my laptop.
4. The Arduino can set a LED, in order to let me know it receives the stuff I send to it wirelessly (and because you have to have a LED)
5. I can send AT-commands to the XBee EndDevice through the Arduino on COM5.
6. The Arduino will wake the XBee up, so that it will Obey My Command.

Here's my Sketch:

/*

Hardware-mods:
- Pin 9 on XBee connected through 1.4Kohm resistor to pin 7 on Arduino (Any pin, allows arduino to control sleeping on XBee)
- XBee RX connected to pin 19 (RX1) on Arduino (Allows Arduino to phone home on USB-serial, for easy debugging)
- XBee TX connected to pin 18 (TX1) on Arduino (Ditto)

XBee config-changes (from default):
SM=5 (If not using the Sleep_RQ pin, set SM=0, and use Router firmware)
D7=0 (CTS on XBee is hardwired to reset pin of Arduino. When XBee sleeps, Arduino resets.)

My setup
Coordinator connected to COM5 on USB XBee Explorer
Arduino connected through USB to COM4, XBee EndDevice on XBee Shield.

Why?
To play with the EndDevice XBee and try to work out what's going on! Send H or L through the XBee connected
to the XBee Explorer (i. e. Coordinator) and watch the LED change, as if by magic! Send AT-commands through
the COM-port connected to the Arduino to see what the EndDevice is playing at, reconfigure parameters etc.
*/

const int ledPin = 13;      // the pin that the LED is attached to
const int sleepPin = 7;     // Connected to SleepRQ on the XBee, SM=5

int incomingByte;           // a variable to read incoming serial data into

int graceperiod = 5000;     // Milliseconds to wait before allowing XBee to sleep, to make sure no more data incoming on USB serial
unsigned long timeout = 0;
boolean sleeping = true;

void setup() {
  // initialize USB serial communication:
  Serial.begin(9600);

  // Initialize XBee comms
  Serial1.begin(9600);

  // Say hello.
  Serial.write("Hello.");

  // initialize the LED pin as an output:
  pinMode(ledPin, OUTPUT);

  // Pin7 now connected to Sleep_RQ
  pinMode(sleepPin, OUTPUT);

  // Xbee sleep
  digitalWrite(sleepPin, HIGH);
}

void loop() {

  // see if there's incoming serial data from XBee:
  while (Serial1.available() > 0) {

    // read the oldest byte in the serial buffer:
    incomingByte = Serial1.read();

    // Echo back to USB
    Serial.write(incomingByte);

    // if it's a capital H or h, turn on the LED:
    if ((incomingByte == 'H')  (incomingByte == 'h')) {
      digitalWrite(ledPin, HIGH);
    }

    // if it's an L or l, turn off the LED:
    if ((incomingByte == 'L')  (incomingByte == 'l')) {
      digitalWrite(ledPin, LOW);
    }
  }

  // Data from USB is echoed to XBEee
  while(Serial.available() > 0){

    // If sleeping, set sleepPin high and wait for XBee to wake up
    if(sleeping == true){
      digitalWrite(sleepPin,LOW);
      sleeping = false;
      delay(500);  // Pick a number..
    }

    // A character was received, renew timeout
    timeout = millis() + graceperiod;

    // Send data to XBee
    Serial1.write(Serial.read());
  }

  // Put XBee back to sleep, after graceperiod (to make sure no more keypresses are incoming)
  if((sleeping == false) && (millis() >= timeout)){
    digitalWrite(sleepPin, HIGH);
    sleeping = true;
  }
}

So.. I just got this Arduino..

So, I decided (for no particular reason, I might add), to order a few geek gadgets. Namely, a couple of Arduino's, with all the fixin's. (Mega and 3v3 Pro, XBee's, with the obligatory XBee Shield and XBee Explorer from Sparkfun.)

Good stuff, I really like that I can get straight to programming it (yes, I have been staring at a stupid blinking LED for several days now, wondering quietly what I was thinking when I placed the order..) and I like how I dont have to spend days assembling all the "ancillary" circuits just to get the damn thing working. Most of all I like the fact that it can be programmed with the Arduino software and the bootloader. Which means, of course, that I have a platform for programming microcontrollers with simply a USB-cable and free software. No need to buy (or solder) an expensive (or cheap) programmer, no need to make a Decision as to which architecture to go with. Buy it, and go. Hassle free. Or so I thought..

So for the not-so-good stuff. Full disclosure; I've owned these things for exactly 3 days, so my knowledge of the hardware and software may not entitle me to an opinion.. I have one none the less. I have also next to no experience with electronics. That only puts me right in the target demographic for the Arduino.

From my understanding, the makers of Arduino (and the related projects) put this package together so that people could get started with HW-hacking quickly and painlessly. Which works just fine, up to a point. You see, the arduino GUI and language hides so much of what's going on under the hood that it becomes a pain to map the real hardware, lets start with the AVR1280 datasheet, to the Arduino software.

I quickly figured out that my path of progression would be something like

1. Make the LED blink using the Arduino GUI
2. Move on to AVR Studio and AVR GCC to get more control and understanding of the operation of the chips
3. Use AVR Studio with ASM.
4. Sit back and congratulate myself for understanding the AVR8 architecture, as well as buildtools

That last point would have to be done quietly, lest someone should overhear me. It would not do.

So, as I mentioned, Step 1: blinking LED, CHECK.

Moving on to Step 2 was more of a pain than I imagined. To recap, I have now written the obligatory blinking LED using the actual registers etc in AVR Studio 4, compiled it, and now I want my Arduino to execute it.

But the Arduino GUI gives no clue whatsoever on how the hex-file gets to your board. A little Googling will reveal it uses AVRdude, which sounds just fine. Of course, things arent a simple as that. I ended up digging through the sourcecode for the GUI to find the interesting bits, an lo and behold, you can add "upload.verbose=true" to your preferences file, which will output all the details on how it uses AVRdude! Success! (For the interested, it's in svn/trunk/app/src/processing/app/debug/AvrdudeUploader.java)

Erm.. No. For reasons I don't fully understand, it doesn't work with the stock AVRdude. Take the following example:

avrdude -Uflash:w:Test.hex:i -cstk500v1 -P \\.\com4 -pm1280 -vvvv -b 57600 \
-D -C\Ole\arduino-0017\hardware\tools\avr\etc\avrdude.conf

avrdude resolves to the version bundled with WinAVR, which I've installed for use with AVR Studio. It is pretty much an exact copy of what the Arduino GUI does. Still, it spits back at me

D:\Ole\Avr Studio Projects\Test\default>avrdude -Uflash:w:Test.hex:i -cstk500v1
-P \\.\com4 -pm1280 -vvvv -b 57600 -D -C\Ole\arduino-0017\hardware\tools\avr\etc
\avrdude.conf

avrdude: Version 5.6, compiled on Mar  5 2009 at 09:59:30
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/

         System wide configuration file is "\Ole\arduino-0017\hardware\tools\avr
\etc\avrdude.conf"

         Using Port                    : \\.\com4
         Using Programmer              : stk500v1
         Overriding Baud Rate          : 57600
avrdude: Send: 0 [30]   [20]
avrdude: Send: 0 [30]   [20]
avrdude: Send: 0 [30]   [20]
avrdude: Recv:
avrdude: stk500_getsync(): not in sync: resp=0x00
avrdude: Send: Q [51]   [20]
avrdude: Recv:
avrdude: stk500_disable(): protocol error, expect=0x14, resp=0x51

avrdude done.  Thank you.

Thank you, indeed.. So. Now what? Turns out, someone must have patched something. If you specify the AVRdude.exe that comes with Arduino, it all works like a charm..

\ole\arduino-0017\hardware\tools\avr\bin\avrdude -Uflash:w:Test.hex:i -cstk500v1 -P \\.\com4 -pm1280 -vvvv -b 57600 -D -C\Ole\arduino-0017\hardware\tools\avr\etc\avrdude.conf

Is this behaviour documented? Quite possibly. It didn't show up in the first 10 results from Google (that I checked). My current guess is that is has something to do with DTR resetting the Arduino. If I use stock AVRdude, and reset the Arduino at precicely the right moment, it will load correctly. Topic for another post..

So, thats one hurdle overcome. Strangely, I couln't really find a cookbook recipe on how to use AVR Studio 4 with the Arduino.. Here is one.

1. Add "upload.verbose=true" to the preferences file of a working Arduino. (This must be done with the GUI shut down, or it will be overwritten)
2. Observe the output
3. Write and compile your program in AVR Studio
4. Upload, using the AVRdude from the Arduino, and using the options observed in step 2.
5. Rinse and repeat.