PocketBible – Craig's World

Working Without a Net

In 1988 we took our kids to Circus World in Baraboo, WI. We had the good fortune to see Enrico and Debbie Wallenda performing on the high wire. The crowd was small, so we had a chance to chat with them after the show.

I noticed that, while they performed without a safety net, they had laid out some tumbling mats under the wire, which was some 20-30 feet in the air. “Why bother with the mats?” I asked Debbie. “Our insurance company won’t let us perform without safety equipment,” she said.

Enrico and Debbie come from a family of circus performers that goes back to the 17th century. They grew up on the wire. They know what they’re doing. The safety net is unnecessary. It would just slow down their set-up and be one more thing they’d have to carry from show to show. And it would be an insult to them. Amateurs need and use safety nets. The Wallendas just don’t fall.

Well, they do, but no more than once each.

I’ve been programming since about 1974 and doing it professionally since 1982. I remember my ham radio buddies talking about these cool new things called “microprocessors” in 1972, and was actively programming for them through the entire history of the “personal computer revolution”. I wrote a machine language debugger for the Motorola 6502 just for fun back in the early 80’s. I was working with touch screens in 1982 while the company that would eventually put one on an iPhone was just beginning to experiment with mice and graphical user interfaces. I wrote drivers that allowed CP/M systems to store data on “magnetic bubbles” that raced along a track driven by an electromagnet. I was writing code to calculate position using GPS when there were only about 8 GPS satellites in orbit and we had to calculate what ridiculous time of the morning we’d have to come in when we would be able to see the required minimum of three of them at the same time for testing our software.

Over the years I’ve programmed in assembly language, Pascal, PL/I, Modula 2, FORTRAN, COBOL, SNOBOL, LISP, BASIC, RATFOR, C, Forth, JOVIAL, Ada, C++, NewtonScript, PHP, Objective-C/C++, Perl, Java, JavaScript, C#, and VBScript. I’ve written a number of compilers and interpreters for lesser-known and proprietary languages. I’ve written programs for Atari, CP/M, MS-DOS, VMS, Windows (desktop and server), Mac OS and OS X, Newton OS, various flavors of Unix, Windows CE, Palm OS, Android, and iOS.

When I’m making updates to the scripts and pages on our Laridian website, I operate directly on the live server. From time to time that means if you happen to be on the page I’m on when I make a mistake, you’ll see an error message when you load the page. But that happens only rarely. When working on a critical part of e-commerce code, I might test on a private page before going public, but often that isn’t practical.

Running a beta testing program with outside testers, especially for a platform like iOS (which requires encrypting the app in a particular way that allows only selected devices to run the program), is difficult and time-consuming. So most of the updates I do for Mac and iOS programs are just tested by me before they are shipped. Often, I’m a better tester than real customers, because I know the scope of the changes I’ve made to the code, and I know where it is susceptible to error. For example, all the testing of synchronization of user data was done in-house. I was able to set up both common and unusual situations to verify how the system would handle the data without putting any real customer data at risk. When the time came to put that system online, it just went online without any beta testing.

So I sympathize with the Wallendas. That safety net takes a lot of time to set up. We know we’re not going to fall. And it takes away from the perception of professionalism — anybody can walk a tightrope when there is no harm in falling.

But when we fall, we fall hard.

Last weekend I shipped an update to PocketBible for iOS that was not tested by outside testers. The majority of the changes were to accommodate the larger screen of the iPhone 6 and 6 Plus. I have an iPhone 6, and I can emulate the 6 Plus in the development environment. Those changes were well-tested.

The other type of changes were related to what happens when you change the password on your Laridian account. Previous versions of the program assumed you were logging into a new account when you entered a different password, so they blocked you from sync’ing your notes, highlights, and bookmarks with the server (so as not to corrupt the “other” account’s existing data, nor corrupt the data on the device with data from the “other” user). Furthermore, they would delete all your books (assuming, again, that you are a different user with a different library of books). The new version of the program records the Laridian customer ID you’re using while sync’ing your database with the server, and only complains about your user data if, when changing your password, you log into a different customer account. Furthermore, it doesn’t delete your books. (I plan to handle that a different way in future releases.)

So while testing the screen-size changes on the iPhone 6, I also tested the new way of determining if you had logged into a different account. It worked great. What I didn’t count on was that Apple had completely changed the way certain user interface objects worked on the iPad sometime in the year since I had issued the last PocketBible revision. As a result, iPad users couldn’t update their data to support the new customer ID tracking method. My testing on the iPad consisted of bringing the program up and trying a few things to make sure it wasn’t completely broken. Since the bulk of my changes were related to laying out the user interface on the larger iPhones, my testing focussed on making sure I didn’t break any layout code on the iPad. I already knew the customer ID tracking worked from my tests on the iPhone.

I joked with one long-time PocketBible customer that I didn’t want to take time to do a beta because I knew he would find problems that I would have to fix. So I shipped version 3.2.0 without the benefit of his help.

Once Apple approved the new version and I installed it on my iPad and saw what was happening, I knew I was in trouble. I began thinking about how those tumbling mats were going to feel when struck from this height.

When Karl Wallenda fell to his death from 120 feet, the family went ahead with their show the following weekend. That’s what professionals do. In my case, I admitted to the mistake immediately, pulled the app from the App Store, posted prominent notices on our blog and on Facebook, and went to work to find the problem. By 2 AM I had the first round of fixes made to the code. After a short nap, I found and fixed the rest of the problems (which required examining about 300 places in the code) and was ready for beta testing by mid-afternoon. We went through three beta versions in 24 hours, and I didn’t upload the final build to Apple until I had a clean bill of health from the testers.

Apple cooperated by approving my request for expedited review, and accomplishing that review within an hour.

In the end I was able to stand up and walk off the mats under my own power. The fans were great; they encouraged me by applauding through the whole thing. I was concerned that some would want their money back, but I got not a single complaint.

While I probably won’t be shopping for a safety net, I may consider doubling the mats.

Advancing Technology by Lying to Developers

Mobile devices have been increasing in screen size, screen resolution, memory, and other capabilities on a continuous basis from the time I got my first Apple Newton MessagePad in 1993. Back then screens were about 336×240 pixels, and each pixel was either on or off — no color. There was a total of 4.625 MB (that’s MEGA bytes) of memory. My first Windows CE device was probably my HP 620 LX in 1998. It was a “clamshell” design with a 640×240 screen and 16 MB of memory.

The thing we knew intuitively from being involved in personal computing since there was such a thing as personal computing was that “change is the status quo”. Our programs never assumed how big the screen was, because we knew our program would need to run next week on a bigger screen, so we wrote our code so that it queried the operating system to ask how big the screen was before dynamically laying out its user interface to fill all available pixels. We never assumed that devices would always be monochromatic, so we wrote our compressed file format to accommodate “words of Christ in red” before they could even be displayed in anything but black on a greenish screen. And even though the entire Bible wouldn’t fit in memory of those first devices, we plowed ahead with the best compression we could manage and a user interface that supported displaying two Bibles simultaneously, knowing that very soon you’d be able to get not just one of our Bibles but two whole Bibles onto the device at the same time.

Fast forward to the iPhone in 2007. When you work for Apple you apparently get big-headed and begin to think you’re among the smartest programmers in the world. Nobody can match your brilliance. Each generation of device you work on is “magical”. It has capabilities and features that nobody could have imagined even six months ago. Features like a more memory and a bigger screen.

Since you couldn’t imagine those features last year, and since you’re God’s gift to technology, you’re positive that nobody else could have imagined those features. So what’s going to happen to all those apps written by people “too dumb to work at Apple” when your new device with a bigger screen comes out? Why, they’ll crash, of course.

Not PocketBible.

You only have to be in this business a week to realize that you can’t hard-code your program to assume a particular screen size. But Apple does this with every single device. Up until iOS 8, we had to prepare a “splash image” to display when the program launched in every possible size and resolution. Currently, that means we have to create launch images in 13 different sizes, one for each iPhone screen size that has ever been shipped, in both portrait and landscape orientation.

Current iOS launch image requirements

If instead they allowed us to manipulate a single image at run-time, we could do all of these with one PNG. But they require us to know every size of every screen we might ever run on (by the way, the image above omits devices prior to the iPhone 4, which would add another half-dozen sizes if they hadn’t already been abandoned by Apple).

This isn’t about managing lots of images. It’s about a philosophy that can’t think past yesterday.

Because of this philosophy, when a bigger screen comes out, Apple either “letterboxes” old apps (putting black bars in the empty space that the program couldn’t possibly imagine would ever be there) or scales them (allowing them to believe the screen is no bigger than last year’s device, then scaling up everything they draw to fill the bigger screen). They believe they are saving developers from having to re-release their apps every time a new device comes out. But in reality, they are requiring every developer to re-release their app to jump through whatever hoop is required to get Apple to stop letterboxing or scaling their apps.

Pre-iPhone 6 version of PocketBible on the left gets scaled up. Adding a “launch screen” (which is unrelated to drawing text) tells iOS not to lie to us about the screen size, producing the sharper image on the left with absolutely no changes to PocketBible code!

With iOS 7, there was a special checkbox we had to check to tell the OS that we understood their new semi-transparent user interface elements. With iOS 8, in order to convince iOS not to scale your app (producing blurry text), you have to provide a special, scalable launch image that works on any screen size. (Gee whiz, 2015 and we’re finally recognizing that screens might get bigger in the future! Thanks, Apple!) Until you do that (which requires re-releasing your app), iOS will lie to you about the size of the screen then scale your user interface up to the bigger physical size of the screen, producing blurry text.

Oh, and you still have to provide those 13 launch images for older devices.

The result of this policy of “technical advancement by lying to developers” is that instead of one guy at Apple having to write zero lines of new code, hundreds of thousands of developers have to update and re-release their apps. There would not have been a personal computing revolution in the 80’s and 90’s if Microsoft would have taken this approach. Back then, Microsoft would collect commercial software products and use them for regression testing of new versions of DOS and Windows. After all, you wouldn’t want to do something stupid and break every single app the way Apple does with every release of the iPhone.

This industry used to be exciting. I was like a kid in a candy shop. Technology was changing and we were riding the “bleeding edge”. Now I feel like the only grown up in the room. I want to slap some of these Apple and Google kids around and tell them to shape up.

A Customer Wants to Know: How Stupid Can I Be?

Back in the day, we used a number of email lists run by a program called mailman to communicate with our customers. You could join a list based on the type of device you had, and from time to time we’d email you to let you know about updates and upgrades. We stopped using these lists around 2007 but the server is still running.

Every month, the list server sends each member his password and a reminder that he can unsubscribe or change his preferences by logging into the server and making the changes. When you sign up for the list, you can turn this option on or off. Because so many people sign up by email and have a password generated for them automatically, this behavior (monthly reminders) is turned on by default.

On January 1 I received this email from a subscriber to our iPhone list. I’ve changed his name and anonymized his employer’s company name, which appeared in every email he sent. Note he’s writing from the UK.

You have just emailed me my user name and password in an e-mail in plain text.

Are you stupid or something!

I have closed my account

Jimmy McWeenie

Jimmy’s Employer’s Name Here

Normally, I would send a nice response that explains that there is no financial or personal data exposed by the password, and would explain why we enable this behavior by default. But his “are you stupid” comment irritated me. I crafted a number of more- and less-tactful responses to this email, but ended up sending this one:

On 01/01/2014 15:09, “Craig Rairdin” <[email protected]> wrote:

Jimmy,

When you signed up for this email list, you chose the option to have the server send your login credentials to you every month. We are stupid enough to send you the information that you requested on the schedule that you requested.

We’re also stupid enough to send you the products you purchase and stupid enough to respond to your support requests. We’re stupid enough to continue developing new products for new platforms and stupid enough to give them away for free.

I hope we’re stupid enough to explain this clearly.

Let me know how much more stupid you need us to be.

I hope your new year is off to the same great start that ours is.

Craig

Jimmy replied:

If you still don’t get that sending someones login details, their user name and password in plain text in an e-mail is not just stupid, it’s a breach of the Data Protection Act 1998, then you should be involved in the computer business at all.

You don’t send this data out every month, just four times since 2011, which was when I had a look at your software.

I’ve done my best to ensure that my account with you is now closed, hopefully be can now both enjoy a 2014 equally undisturbed by each other

Best wishes

Jimmy McWeenie

Jimmy’s Employer’s Name Here

I had to look up the “Data Protection Act of 1998”. It was at this point I realized Jimmy is in the UK.

On 1 Jan 2014, at 22:54, “Craig Rairdin” <[email protected]> wrote:

Jimmy,

Our company and our server is in the US. We haven’t been subject to the laws of the UK since the late 1700’s. 🙂 This is an email list you signed up for. When you signed up, you had the option to have your login credentials sent to you every month. You chose that option. The list server is following your instructions.

Every mailman list server list from the beginning of the internet has done this. I get these reminders every month from a dozen lists. I’m glad to hear you figured out how to remove yourself from the list, which is one of the options that is presented to you every month. You have not “closed your account” — just removed yourself from an unused mailing list.

I’m working on that particular server today and will shut down all the lists while I’m there. We haven’t made use of them for a long time and most people have removed themselves already.

Craig

Jimmy replied:

Unfortunately Craig, you are wrong again. Your company is currently offering it’s products through Apples UK App Store, and so those accounts will be liable to UK taxes and jurisdiction.

I very much doubt that any one who signed up, expecting some kind of news letter, thought that you would e-mail out their account details in plain text.

Tell you what, I’ll e-mail our conversation around to a few websites tomorrow, and we’ll see if, generally, people think that your company is behaving irresponsibly or not.

I’ll cc you in so that you can know who I’ve sent them to, as no doubt they will want some comments from you

Regards

Jimmy McWeenie

Jimmy’s Employer’s Name Here

So now Jimmy is threatening to expose this vile breach of privacy to the rest of the world. My experience is that people who make this threat either never follow through, or else the people they notify are used to receiving their crazy rants and just block them. So I’m not worried. Continuing to demonstrate the scope of my stupidity, I chose to respond:

On 2 Jan 2014, at 00:00, “Craig Rairdin” <[email protected]> wrote:

Our relationship with Apple is one where they act as an independent seller of our software. Our agreement with them makes them responsible for all taxes and local laws in the places in which they do business. It does not create nexus in the UK for Laridian. In fact, one of its purposes is to assure us that it is Apple that is doing business in the UK, not us. The people who signed up for the list learned about the purpose of the list on the same page where they opted to have their password emailed to them every month. If they knew they were signing up for a newsletter, they knew they were requesting their login credentials. And if they objected to receiving those, they read the instructions and learned, just as you did, how to remove themselves from the list or change their subscription settings.

Feel free to pass our conversation around to whomever you feel it would benefit. Make sure to let them know that we sent you the information you requested, that we told you how to stop receiving that information, and that you followed those instructions and now are not receiving that information any longer. If that angers them to the degree it does you, I’d be happy to discuss it further with them.

Craig

This morning, Jimmy replied:

On 1/1/14 6:13 PM, “jimmy mcweenie” <jimmy@jimmys_employer.co.uk> wrote:

I’m sure that Apple will be one of the people I send this to. From a brief viewing of Apples terms and conditions, it would seem to me that they make some effort to preclude the type of liability you suggest falling on them. Would you like to take the opportunity, right at the start, to send me a copy the details of where I signed up to have my account information sent to me in plain text? You seem to want to rely on the fact that I asked you to do this, and you were only complying with my wishes.

If you send me evidence that I specifically asked you to email me my account security information as plain text each month, I will include this information with my email of this conversation.

Is there anyone at Laridian you would like to involve in this discussion?

Regards

Jimmy McWeenie

Jimmy’s Employer’s Name Here

By threatening to involve other people at Laridian, he’s hoping to get me worried that my boss will find out how I’ve been treating our customers. Clearly, Jimmy hasn’t read the Laridian org chart. When it comes to stupid, I’m the top dog here. I decide to bring this to an end.

On 02/01/2014 16:40, “Craig Rairdin” <[email protected]> wrote:

Here’s the documentation for Mailman, our list manager software: http://www.gnu.org/software/mailman/docs.html. It contains everything you need to know.

You may involve as many people in this discussion of the subtle details of your email preferences as you think will be interested. Personally, I’ve lost all my interest. I have explained the situation to you; you have removed yourself from the inactive list you signed up for; I have removed you from any future email we may do (though we probably haven’t sent you a marketing email in the last 10 years or so); and I’m in the process of shutting down this unused mailing list server. I’ve disabled the automatic monthly emails, which is irrelevant since I plan on having the entire physical server offline in the next few days.

I have explained that we’re not subject to the Data Protection Act of 1998 since we have no nexus in the UK. Furthermore, the mailing list does not retain any “sensitive personal data” as defined by the Data Protection Act of 1998, so unauthorized acquisition of your password would not expose any data that is protected by the Act, even if we were subject to it.

This will be my last email to you on this subject, which, as you so tactfully put it, is “How stupid can (I) be.” I believe I answered that question by simply replying to your email. I made it clearer by continuing the conversation as long as I have.

Again, I wish you the very best for the new year.

Craig

But Jimmy will have none of it. He continues:

Craig

In a mission to prove how stupid you really are, you decide to have one last poke at me, when I hadn’t taken my complaint any further.

I had decided that in light of the fact you sold Bible software, I would put the exchange to a down to a bored techie whiling the time away making ill advised comments to a customer.

However if you want to discover how serious this complaint is, I’m e-mailing Apple to see what they think of Laridian distributing customer account details across the weld every month

Enjoy

Regards

Jimmy McWeenie

Jimmy’s Employer’s Name Here

So sometime between when he said he was “sure” he was going to send this all to Apple and when he received my reply, he decided not to. Only when I replied did he decide to cry havoc and let slip the dogs of war. Interestingly, my reply contained no “pokes” at him, only a continuation of the self-depricating theme of answering his question “How stupid are you?” in the affirmative.

I’m sure since Jimmy has no idea that I’m the President of Laridian that he has no idea who to send his email to at Apple. I’ll let you know what happens next.

Implementing Interprocess Locking with SQL Server

I suppose everyone does this and I just haven’t heard about it. I don’t get out much, so it seems cool to me.

When we redesigned our company website (www.laridian.com) a couple years back, I needed a way to automatically update best-seller lists, new releases, and other dynamic data on the site without relying on an employee to do it every week/month/quarter. Initially, I considered writing a script that did this kind of thing and was launched by the OS on a schedule every so often, but I try to stay away from creating yet another little thing I’ll have to remember if we ever move the site or are forced to recreate it on another server.

So it occurred to me that I could keep track of when the last time was I had created a particular list or other piece of dynamic content on the site, and the first user who requests it after some time period (say once a month for “best sellers” and once a week for “new releases”) would cause the site to notice the content was old and regenerate it. That’s a cool idea on its own, but isn’t the subject of this article.

One of the problems I wanted to avoid was having two or three users who happened to show up at about the same time all trigger the process. I was concerned that it might be time-intensive and while I don’t mind delaying one customer while the data is created, I didn’t want to delay everyone who visits the site during those few seconds. So I came up with the idea of using SQL Server to implement a generic “lock” or “semaphore” capability I could use anywhere on the site.

The idea is to have a simple table with a Name field and a SetTime field. The Name field is given the UNIQUE constraint, so that duplicate records with the same Name field are not allowed. The first customer session that discovers it needs to rebuild the best-sellers list tries to INSERT a record with Name = ‘Best Sellers’ and SetTime = GETDATE(). If the INSERT succeeds, the process “owns the lock” and can do what it needs to do. If someone else comes along shortly thereafter and discovers it, too, needs to update the best-sellers list, it will try to do the same INSERT and will fail due to the existence of a record with the same Name field. This second process does not own the lock, and cannot update the best-sellers list. Instead, it uses the old list.

Once the first session has updated the list, it simply DELETEs the record, thus releasing its lock on the best-sellers list.

Since INSERT is an atomic operation there’s no possibility that two sessions are going to both believe they wrote the record.

Since the web is a flaky place, it’s necessary to allow for the possibility that a lock obtained a long time ago was never released. So every request for a lock checks the SetTime field. If the existing record is “too old” it is deleted before the attempt is made to INSERT the record.

This allows a certain amount of interprocess cooperation and communication between my Classic ASP pages with very little effort.

One of the side-effects is that the locks span not only all the processes running on the server, but can be made to span processes running on user devices. A recent use case that surfaced for this capability was the necessity of keeping a user from synchronizing his notes, highlights, or bookmarks from two (or more devices) with the Laridian “cloud” at the same time. The results can be unexpected loss of data on one or both of the devices.

The solution to this potential problem was for the synchronization process to request a lock that contains both the name of the table being synchronized and the customer ID. That way, many customers can synchronize, say, Bible bookmarks at the same time, but any one user can only synchronize one device at a time. This is a little more complicated than it seems, since PocketBible for Windows and PocketBible for iOS each have their own synchronization script on the server, while our newer clients (PocketBible for Android, Windows RT, and Windows Phone) use our new TCP-based synchronization server. The scripts for the older clients are written in Classic ASP and are invoked through HTTP POST operations from the client, while the new TCP server is written in C# and runs as a Windows Service. All have access to the same SQL Server database, and all implement the same locking strategy, which is working well.

In addition, during the debug process the TCP server runs on my local machine and connects via VPN to SQL Server. I can use and test the locking mechanism in this way before it goes live.

The combination of a very simple implementation using technology (SQL Server) that is well-known and well-tested, and the ability to implement locking across platforms makes this an interesting and (I would argue) elegant solution to a large number of problems.

PocketBible 2 for iOS

We’re just about to release an update to PocketBible for the iPhone/iPad. This version adds some often-requested enhancements and some new features that you probably didn’t even know you needed.

First, we’ve added the ability to select any text and copy it to the clipboard (what the iPhone calls the “pasteboard”). You’d think this would’ve been a standard feature of the program from a long time ago, but since we wrote our own HTML rendering code, it all had to be implemented by hand — including the cool magnifying glass that shows you what you’re touching as you select text and the lollipop-shaped “drag handles” that let you expand the selection.

Next, the multi-paned user interface was enhanced to allow you to represent each pane as a tab. This lets you easily jump from a Bible to a commentary and back by just selecting a tab. This feature was further augmented with the ability to open all your books in one operation. Your books will open in five tabs/panes labelled “Bibles”, “Commentaries”, “Dictionaries”, “Devotionals”, and “Other”. If you don’t have any books in a particular category, that tab/pane is not shown.

For those who take notes in the expanded toolbox, you’ve been frustrated by the fact that expanding the toolbox covers any open books beneath it. As a result it can be hard to both follow the Bible text and take notes (or preach from your notes). We’ve modified the behavior of the toolbox so that it switches you into tabbed mode if you’re not already there, then moves the current book out of the way of the toolbox. Now your Bible can be displayed in a narrow column above, below, or to either side of your notes. Very nice.

Two major new features are introduced in version 2. First is what we call “Autostudy”. By simply selecting a verse (or a word), you can quickly get a report showing everything your library has to say about that verse/word. While Autostudying a verse, you’ll see that verse in all your selected Bibles, the verse with Strong’s numbers if you have one of our Bibles that contains Strong’s numbers, definitions of every word in the verse from your selected dictionaries, definitions of all the Strong’s numbers in the verse from your selected Strong’s dictionaries, and all the cross-references for the verse from the Treasury of Scripture Knowledge. You can select which books and Bibles are used by this feature and in what order the material appears.

When Autostudying a word, you’ll see how many times it occurs in each of your Bibles, what Greek or Hebrew Strong’s words are translated to that word, and the definition of those Strong’s words and the selected word from your dictionaries.

Autostudy reports can be browsed in PocketBible, saved to PocketBible’s public folder (and from there transferred to your desktop computer through iTunes), copied to the clipboard, or printed to an AirPrint-compatible printer.

Finally, PocketBible now has the ability to speak any Bible or reference book you own. Simply purchase a voice (eight voices in two languages are available) using In-App Purchasing in PocketBible or at our website, then select the text you want to read. You control the volume and can even have PocketBible read while your iPod music plays in the background. You can set PocketBible to start reading at a particular verse and just keep going until you tell it to stop, or give it a specific passage to read.

The basic PocketBible program with the same features as the current version (1.4.7) will be free. The advanced features described above will be available for a nominal fee, either as an In-App Purchase or at a lower price at our website.