Prev
Next

Virgin blip gives a glimpse of total digital chaos

Paul Colgan

by Paul Colgan

27 Sep 10:20am

Must Reads

The Punch is moving house - this is a caption

The Punch is moving house

385

Will Pope Francis have the vision to tackle this? - this is a caption

Will Pope Francis have the vision to tackle this?

122

Advocating risk management is not “victim blaming” - this is a caption

Advocating risk management is not “victim blaming”

251

Filed under: Air Travel, Computers, Cybercrime, Internet, Tech, Technology, Travel, Virgin Blue

43 comments

It’s a very first-world picture of human misery: a packed airport terminal filled with thousands of delayed travellers.

The chaos yesterday ... and that's just Sydney. Picture: AAP

There are frazzled parents at the limits of their patience, looking after bored kids giddy at being on their school holidays but frustrated at having nothing to do. Passengers milling around, trying to nap on a hard floor, anxious that the next announcement on the public address system will be the one that cancels their flight.

And all because of a computer problem.

It’s the second edge of the sword of connected technology: systems that make aspects of life easier, cheaper, faster and more convenient – until they run into a problem.

If a system crash that affects one airline can result in this kind of chaos, consider the chaos if something bigger failed, like Centrelink, or a bank.

A simple crash or a cyber attack on any number of systems could quickly bring the economy to its knees – which is why they are such appealing targets for terrorists.

The US Secretary of Defence Robert Gates said last year that his department was now dealing with cyberattacks every day and he planned to quadruple the number of staff protecting the critical systems from being compromised.

Two years ago, a whole network of US defence computers was infected by a virus that allowed hackers to steal information, simply by a little flash drive – the little things that go on your key ring – being plugged into a computer somewhere in the Middle East.

It wasn’t anything as malicious or premeditated as this that brought about the chaos in airports around Australia yesterday.

Virgin Blue said its external supplier of the reservations system, Navitaire, “had a computer hardware failure that forced the switch to the slower manual check-in system.”

Further details are yet to emerge but “computer hardware failure” sounds to me like corporate speak for a box breaking down or a cable being severed or pulled out of a wall.

One of the key arguments advanced by the government in favour of the National Broadband Network is that connected technologies will create a new virtual economy, unleash new opportunities for business and improve government services, particularly in healthcare.

Which is all very agreeable, but increasing dependence on these technologies creates new risks.

NBN or no, governments and private companies will spend big to try and avoid their systems being compromised but as their rate of use increases so to does the likelihood that there will be a “computer hardware failure”.

While problems with traditional infrastructure that helps us get on with things: water and electricity supplies, transport networks - tend to be confined to a physical area, they can still cause complete chaos. Think of a power cut or a major traffic jam; it’s a massive inconvenience but the impact is largely contained.

But as the Virgin Blue incident showed the effects of a system problem can instantly be felt right around the country. 

It’s not a matter of if a more wide-ranging failure occurs, but when. Not to be a doom-sayer, but the potential chaos from a major system collapse makes the Virgin incident look like a holiday.

Most commented

43 comments

Show oldest | newest first

    • dovif says:

      10:46am | 27/09/10

      Virgin ... they know customer service

      Reply

    • Justin says:

      11:04am | 27/09/10

      Any big business worth it’s salt has redundancies in place to cover just such an emergency. As such, these things will continue to happen, but the impact will always be very temporary.

      Reply

    • Rich says:

      03:08pm | 27/09/10

      Typical end-user attitude. ‘this shouldn’t happen’, but it can and does happen, even with the best planning.

      While it might be true that redundant systems will mitigate the impact of component failure, it is harder if you’re dealing with a soft-failure (also known as a brown out). In this case the failure may take time to be detected by a skilled operator. Data corruption is also a major issue.

      Redundancy is non-trivial, and as Paul points out, the price of a connected culture is that these events will continue to happen.

    • Mr Subramanian says:

      03:51pm | 27/09/10

      Additional problem: redundncy systems cost money to install, to maintain, and to test. Organisations willing to spend the necessary money and time to get this right are normally ones who have experienced the pain first hand. But the profit growth and return on investment criteria are always going to apply pressure, just lilke it does to other safety systems, like aeroplane maintenance.

    • acotrel says:

      07:03am | 28/09/10

      In any given situation there are four areas of risk which must be managed - Quality, Safety, Environment, Security.  The balance between these must be maintained and controlled.  The Quality aspects are about gaining repeat business.  I suggest Virgin Blue will have a problem even keeping its customers. The system should have been failsafe, what does it say about the rest of the organisation?

    • Scot says:

      11:42am | 28/09/10

      Justin you are 100% correct. This is total ineptness on the part of Virgin and its so called supplier of services. This should be a fully redundant system in systems and network. For this system to go down twice demands a complete audit by experts that know about fully redundant bank, insurance and airline systems. It seems to me on the surface they have implemented system changes and have failed to ensure they have a a backup system and not practices backup systems as well.. The big issue today is people have MS tattooed on their backsides and have thrown out the rule book on system changes and upgrades. They think it is OK for a system to fail. Well, in the 70 and 80’s this would never happen, this is outright stupidity and gross incompetence and the end cost is a major impact to the enterprise, heads should role.

    • bella starkey says:

      11:08am | 27/09/10

      As a wise man once said:
      The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.

      Reply

    • Gregg says:

      11:54am | 27/09/10

      It is interesting that the NBN and low carbon are claimed to be the ways of the future and all that is doing is seeing the eye being taken of the ball of the real power houses and their gradual demise.
      Ironically it is the climate change and carbon reduction pundits that have not only governments frozen in their approach to planning for electrical power but hallucinations on how non renewables will save us.

      There was a time when state governments bore the brunt of planning for growth in power demands but then along came many reforms supposedly to improve the system and for the bulk of the population on the east coast this meant an interconnected power grid and a sharing of generators to some extent and that in allowing for management of plant outages over a much bigger area gave some economies of scale.

      The other side of that coin was individual power stations were run for longer and now in private hands are being run harder at peak demand times for that is when the generators earn more per MW and along with power stations not getting any younger, a nominal 30- 40 year life span and a semi-national system that sees no state decisions being too readily taken, great power demands for desalination plants and even the Queensland Premier having said that no new coal fired power station will be built unless CO2 can be stored underground [ a long way off if ever ] the failure of an airlines booking system will pale into insignificance when the eastern states power grid is stretched to its limit to provide and it may not just be for a specific area Paul.
      Do some homework on demand growth Vs current/new generation capacity and you might see that whereas we have a federal government cobbled together with the greens and a medley of independents the power generation system is becoming something similar with a medley of small gas turbine plants here and there and just as the mix of parliament does not offer a lot for the basis of government, nor does the use of band aid solutions offer too much for our base load power requirements.
      Power failures will mean computer systems go down and more for the water pumps need power too, not to mention those desalination plants, the NBN, hospitals, train services, industry and businesses.
      Check out what happens at the check out in a supermarket next time the power goes off!
      And we have $43B+ to blow on the NBN and solar and wind are going to get us home!

      Reply

    • Shane From Melbourne says:

      01:21pm | 27/09/10

      Where’s Enron when you need them?

    • Richard says:

      03:36pm | 27/09/10

      I have wondered if all the hundreds of thousands of tons of ore, coal and minerals etc. that get mined in the southern hemisphere: Australia, Africa, Brazil etc, and shipped off to the northern hemisphere is actually changing the balance of the mass of the earth, causing it to wobble in its rotation more than normal, and is contributing to the changes in climate that are happening in our time?

    • HappyCynic says:

      10:00am | 28/09/10

      @Richard

      Lol, could you be any sillier?  Do think removing a few million tons from the surface of the earth could actually change the rotation of the planet?  The Earth weighs in at 5,972,000,000,000,000,000,000 metric tons (5.972 sextillion tons), even if you were to remove 10 trillion tons of dirt (that’s 10 with 12 zeroes) and put in the Northern hemisphere you’d be only shifting 0.000000167% of the Earth’s total weight.  That’s like shaving one gram off of a 6 million ton ball of iron and asking someone to tell the difference in a before and after shot.

      Besides earthquakes shift more weight around in a single minute than we can mine out of the ground in a decade, then there’s plate tectonics

    • nosthow says:

      12:03pm | 27/09/10

      Well you get what you pay for dont you ? Virgin trim costs everywhere in pursuit of dollars and it seems even their computer system is sub par - wouldnt fly with them myself. They remind me too much of the Coalition and Tony Abbott - all feathers and no bird ! Geez I am on fire today Coalition supporters !

      Reply

    • Phil says:

      12:49pm | 27/09/10

      Australians have a problem where they seem to bring politics in to everything no matter how unrelated, its quite frustrating!!

      Id fly Virgin any day, id rather walk or drive than fly Qantas.

      You are foolish to think that all companies dont try to cut costs where possible, just look at Qantas and their history of servicing and maintenance - cost cutting has caused quite a few problems with them & it will only be a matter of time before it costs someones life.

      I also love Pauls view on “it much be a cable that is unplugged” yeah because thats always what it is, a cable not plugged in. 
      Dont quit your day job and leave the tech work to techs.

    • nosthow says:

      04:11pm | 27/09/10

      @Phil - am a wealthy self funded retiree Phil. No need to quit my day job !  hahaaha

    • Reg says:

      08:31am | 28/09/10

      Hey Phil… here’s a line from the Simpsons when the parking inspectors took the day off. See if you can spot the political message.

      ‘Without the crushing rule of law, society will do a better job of regulating itself.”  wink

      Over to you.

    • tcm says:

      12:16pm | 27/09/10

      Jeez a bit harsh here - I am pretty sure that Virgin didn’t get up yesterday morning with the express idea of having their computor system crash and burn….....as for the frustrated passengers…two words TRAVEL INSURANCE and can anyone tell my why people have the immediate need to ‘nap’ on a concrete floor ??????

      Reply

    • Gregg says:

      01:02pm | 27/09/10

      Is it really a beat up on Virgin or merely an article to say a too heavy reliance on all things digital for the mass speed of society can mean some massive slowdowns too.
      Napping on the floor sure beats standing around when there are no longer any vacant seats and if you want to maintain your queue place in vain hope!
      Travel Insurance is not going to move you though it will help for any connections but then most airlines are not too bad in this sort of situation anyway.
      Other than that, a Plan B can always be handy.

    • fairsfair says:

      01:06pm | 27/09/10

      Totally agree re Insurance. There is always the chance of something going horrifically wrong to ruin your travel. Whether that be just for you - or in this case - a huge chunk of people.

      As with general and vehicle insurance, the common misunderstanding is that people purchase insurance in case they do something that causes them loss or damage. NO - the need is due to some other idiot causing loss or damage to you. Fault is a factor that determines recovery (and only that) in Insurance, so even if you are not at fault - you still have to have the product and lodge the claim (and pay your excess unless your insurer makes the decision to waive it on the basis of the information supplied). Third Party recovery is a whole other beast and unfortunately you have limited rights in that situation. If you don’t have insurance - you are the idiot.

      I commend Virgin for their handling of this matter. It is completely out of their hands and they can’t snap their fingers and fix it to the liking of everyone. They are doing a great job given the circumstance and I will always fly them over Jetstar because whether or not the first comment on this article was laced with sarcasm or not - they do offer good service for the cost of their product. I’ll pay for my coffee and have it served to me by a smiling person over getting a complimentary bag of nuts off some horrid angry old bat on Qantas anyday.

      Re the concrete naps - I keep waiting for someone to sue for “back damages” as a result of the delays.

      Sometimes we all just need a reality check and acknowledge that bad things happen to good people! Those things are often out of our control and are ultimately character building.

      I hope it is resolved ASAP.

    • Bill says:

      10:45am | 28/09/10

      Thank you for using common sense. I used to work in customer service and had to deal with IT breakdowns all the time. Guess what, I didn’t invent/install the system but I always copped the abuse because someone got delayed by 5 minutes. I feel for the Virgin Blue staff who probably copped abuse all day. People need to be mindful of the fact that things do break down and while everyone is abusing Virgin Blue, no one reported the last how many days the system worked fine.

    • iansand says:

      12:18pm | 27/09/10

      Almost certainly Rob Oakeshott’s fault.  Or a Collingwood supporter.

      Reply

    • Smee says:

      05:48pm | 27/09/10

      iansand - Nah, must be Tony Abbot’s fault eh nosthow?

    • nosthow says:

      06:10pm | 27/09/10

      @Smee - great minds think alike buddy !

    • John says:

      12:51pm | 27/09/10

      This is just a poorly managed system, anyone with good IT knowledge would know that a reservation systems is core to the business and highly critical. Hence the disaster recovery and business continuity needs to reflect this. Manual process can work, but ask a VB person how often they practice this, I’d say… none. This is purely a case of badly managed operations. I’d be concerned if these practices make there way over to engineering and maintenance as well.

      Reply

    • Rich says:

      03:14pm | 27/09/10

      Manual processes can’t work anymore. The entire system is electronic.

      If you want to be able to check in up to 30 minutes before a flight you have to make a trade off for the lack of ‘paper backup’.

      It’s not just passengers thats impacted, it’s all of the freight and luggage etc.

    • John says:

      04:53pm | 27/09/10

      Rich, as I said badly managed. I’ve overseen some pretty extraordinary automation pieces of work within the mining industry, and one of the must haves is a back up plan, manual or semi automatic whatever the case must be. These plans are tested, deployed as a monthly training exercise. I don’t believe in the excuse that just because everything is computerised, there is no backup plan.

    • Wombat says:

      08:11pm | 27/09/10

      About 4 months ago I was flying from Jakarta to Kuala Lumpur then connecting to Bangkok. I was standing in the queue at Jakarta airport when the Air Asia system crashed. It’s a bit of a cultural thing there that nobody wants to make a decision (you can see it in Australia too sometimes - as long as you don’t make a decision then you can’t stuff up). After about 30 minutes an Air Asia girl took it on herself to start writing out boarding passes on slips of paper.

      I guess some of her friends must have started writing out boarding passes too (on what looked like baggage tickets) because on the plane a number of seats were assigned to multiple passengers. It all worked out though. I’m sure that it would have been significantly more difficult to do this at an airport that has to pretend to be taking precautions against terrorists, as Australian airports do.

      In KL 3 hours later the system was still down. They had proper boarding passes for the passengers filled out by hand. Queues were short. Everyone survived.

      Maybe we just can’t expect the same degree of efficiency and competence in Sydney as in Jakarta or KL.

    • Rich says:

      11:01pm | 27/09/10

      @John.  I’ll take my experience in providing mission critical services to the airline industry over your mining experience on this one.

      Your comments are correct. Back up systems, redundancy and proper training are all essential to ensuring minimum impact of an unplanned outage.

      However your original comments about manual processes ignore that airlines are now reliant on electronic ticketing. I’m not saying that VB did everything they should have, but let’s wait and see before we take an I know best attitude.

      @wombat sounds you got lucky. Of course you’re comparing two unrelated incidents. Were the outages in the same system? Was the entire airline affected? Where did the passenger manifests come from? A “systems outage” could refer to a very wide range of different problems.

    • Dave Sag says:

      08:09am | 28/09/10

      In truth Virgin’s online checkin has always been rubbish too, and their website is something of a dog’s breakfast.  The whole separation of powers between their airline and their frequent flyer programme too is maddening for the regular flier.  It really seems to me that, early on in the piece, someone at VirginBlue made some truly terrible IT choices and those choices are still biting them in the ass today.

    • Wombat says:

      12:21pm | 28/09/10

      Rich

      The whole point was that I was comparing 2 unrelated incidents. And doing so with very little knowledge of either. I don’t know anything about IT but I’ll try to answer your questions.

      In Jakarta the computers were useless. The girl started writing out boarding passes at a desk that didn’t even have a computer on it. It was just a fold-up desk set up against the opposite wall to the check-in counters. I saw what was happening and left the check-in queue and joined the queue to that desk.

      All that I showed was my printed ticket and the girl wrote out my boarding pass. In Jakarta you (are supposed to) need a printed ticket to get into the terminal building. I saw a girl who just had a booking number get turned away without a boarding pass. There was certainly no checking against passenger manifests (unless they did it later when nobody was looking).

      In KL it was done at the check-in counters so I couldn’t actually see if the computers were turned off. But I did see that my check-in girl had a list of seat numbers on a piece of paper. She crossed mine off as she wrote out my boarding pass. I assumed that the plane’s seating had been divided up and each check-in counter given their own seats to allocate. I didn’t see anyone arguing over seats on that plane.

      I also assumed (from the way it was handled) that this was the same problem in KL as in Jakarta, just that in KL they had implemented a good back-up system whereas in Jakarta they had spent the first 30 minutes hoping that the system would come back on and had then scrambled.

      Of course I can’t be sure that this is actually what happened. But it seems to make sense that places suffer regular computer system crashes (and even blackouts) would have needed to develop back-up systems before or at least would have staff that understand the importance of setting up a bodgy system so that the planes can take off on time.

    • food for thought says:

      12:42am | 29/09/10

      @wombat and @ rich

      to clarify re the similar issues air asia and jetstar in the past both airlines use the same booking platform and have at some degree experienced similar issues/ outages. i remeber when jetstar changed out to the same booking/ reservations platform (navitare) early last year they were without a booking system for 3 days ( their call centre was closed refering guests via a recorded message to phone back later in the week. this was done over a weekend packed with leasure travellers. they had delays and canx flight but unlike virgin did not offer compensation or hotel accomodation ......
      .

    • Rossco says:

      01:02pm | 27/09/10

      It’s like Die Hard 4, without the crappy plot but with all the toned down PG-13 badness.

      Reply

    • GladI'mNotFlyingAnywhere says:

      01:58pm | 27/09/10

      John is right…  this reeks of a poorly architectured system with no redundancy or disaster recovery built in.

      This is one incident I’m glad I didn’t have to manage!  eeeks

      Reply

    • BMJ says:

      03:14pm | 27/09/10

      Sensationalism on a grand scale.

      Virgin has a bad day and suddenly we are told to fear total digital chaos with an army of nerdy jihadists plotting our economy’s downfall. Truly laughable.

      Anyway I’m off to finish building my bomb shelter.

      Reply

    • Hermano says:

      03:43pm | 27/09/10

      Where do you live?  You know, just in case…

    • Shane says:

      04:37pm | 27/09/10

      What does Virgin Blue expect? They use the Navitair system who have partnered with Accenture. Anything Accenture touches costs too much, is ran by idiots and in this case clearly doesn’t have any redundancy in place. What i’d like to know was this outage due to Navitair having no redundancy in their systems or if they do, but Virgin Blue choose not to use it to save money?

      Reply

    • Rich says:

      10:50pm | 27/09/10

      From the reports i’ve read it’s not clear whether the ‘no redundancy’ comments referred to navitair’s systems, or whether this meant that navitair were VB’s sole provider.

    • dirtbiker says:

      05:14pm | 27/09/10

      Computer says “no”.

      Reply

    • MF says:

      05:24pm | 27/09/10

      Glad my flight tonight is on Qantas then!  Sucks to be booked on a DJ flight right now.

      Reply

    • Outsourced says:

      07:39pm | 27/09/10

      Outsourced to the guys that are too busy fixing the games village in Mumbai?

      Bring on child labour to fix the problem - they can program my VCR, so a booking system shouldn’t be too much more complicated…

      100,000 SLF on 100 cancelled flights is 1,000 per flight - that’s packing them in like sardines, isn’t it?

      Whatever happened to contingency, backup, and failover systems?

      Reply

    • GiveMeARest says:

      09:09pm | 27/09/10

      There is no such thing as 100% available 24*7*365. Sometimes the computer needs a rest.

      Reply

    • Reg says:

      08:22am | 28/09/10

      Yair   yair   yair… so what if the total banking interchange system decides to take a rest for a week or a month?  This folks is a grand illustration of what is meant by back-up. Under your bed you keep a bin of money that you keep refilling against the rest period our friend so casually predicts. Or perhaps that bar of gold so you can scrape a bit off to pay the baker for yesterdays weevily bread.

      Do you trust these IT guys? I don’t, and I’m not alone. One technical friend has two computers, one he NEVER connects to the web and another that does. Naturally the first one is covered with Alfoil and shielded against the big magnetic pulse that’s due any day now.

    • stephen says:

      11:13am | 28/09/10

      Glad you said that, and I expect those wanting to roll out the NBN will take another look at a 43 billion dollar lemon.
      (Though I suppose if it’s not communications we spend our money on, some bright spark will pipe up ‘We need new subs’.)

    • Mark says:

      03:57pm | 06/10/10

      Actually, you are all wrong.

      The REAL issue here - is one of corporate governance and due care. Virgin, like many companies today, are all too complacent when it comes to outsourcing. Executives are all too happy to absolve themselves of their corporate responsibilities in favour of contracts and SLAs. They believe that if you have a contract in place that says a supplier must meet a SLA - you have executed your corporate responsbility.

      The Virgin outage is a classic example of where an organisation who outsources a business critical function to a third party failed in it’s duty of care to actively ensure that the third party was capable of and continuing to deliver what was required.

      In this instance, the solution provided by the third party had more redundancy than you could possibly imagine. But poor decision making, planning and communication by the supplier resulted in a longer than necessary outage.

      Further, I question why an organisation would opt for such a cutting edge and largely unprooven enterprise technology like solid state disk storage for such a mission critical application - and what governance Virgin executed in the decision making process for the technology solution - or did Virgin simply let the service provider determine how to deliver the solution as long as the service provider signed up to the contract and the SLAs.

      My view - contracts and SLAs are no replacement for prudence and due care.

      Reply
 
Sorry, comments are now closed for this article.

Facebook Recommendations

Read all about it

Punch live

Up to the minute Twitter chatter

Malcolm Farr

farrm51 25 May 13 09:41PM

RT @CrawfordFund: @farrm51 u may like 2 help spread word of our #foodsecurity journo award http://t.co/FwbMWwJmLf

Daniel Piotrowski

drpiotrowski 25 May 13 12:59PM

RT @adamroy37: Just received a phone call from a young girl apologizing for her actions. Lets support her please #racismitstopswithme#Indi…

tory_maguire

_Tors 25 May 13 12:42PM

RT @adamroy37: Just received a phone call from a young girl apologizing for her actions. Lets support her please #racismitstopswithme#Indi…

Daniel Piotrowski

drpiotrowski 25 May 13 12:03PM

Australia. Where you die for your country and get a rest area named after you http://t.co/hO6LpfwDvI

See all tweets

Recent posts

The latest and greatest

The Punch is moving house

by The Punch Team

385

The Punch is moving house

Good morning Punchers. After four years of excellent fun and great conversation, this is the final post…

Will Pope Francis have the vision to tackle this?

by Hannah Harborow

122

Will Pope Francis have the vision to tackle this?

I have had some close calls, one that involved what looked to me like an AK47 pointed my way, followed…

Advocating risk management is not “victim blaming”

by Sarrah Le Marquand

251

Advocating risk management is not “victim blaming”

In a world in which there are still people who subscribe to the vile notion that certain victims of sexual…

Nosebleed Section

choice ringside rantings

From: Hasbro, go straight to gaol, do not pass go

Tim says:

They should update other things in the game too. Instead of a get out of jail free card, they should have a Dodgy Lawyer card that not only gets you out of jail straight away but also gives you a fat payout in compensation for daring to arrest you in the first place. Instead of getting a hotel when you… [read more]

From: A guide to summer festivals especially if you wouldn’t go

Kel says:

If you want a festival for older people or for families alike, get amongst the respectable punters at Bluesfest. A truly amazing festival experience to be had of ALL AGES. And all the young "festivalgoers" usually write themselves off on the first night, only to never hear from them again the rest of… [read more]

Light weight

Gentle jabs to the ribs

Superman needs saving

Superman needs saving

Can somebody please save Superman? He seems to be going through a bit of a crisis. Eighteen months ago,… Read more

28 comments

Newsletter

Read all about it

Sign up to the free News.com.au newsletter