SpringSource

I’ve just started playing around with Spring Roo it seems like a really good ’scaffold’ approach to starting Java development, much like the productivity boost you get from Ruby on Rails or Grails.  There is a good tutorial here.  I did hit one issue while following it – on Windows.

If you have installed Roo on a different drive than the projects you create, for example I installed Roo to E:\apps\roo and then tried to create a Roo project in C:\rootest and got this:

Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/r
oo/bootstrap/Bootstrap
Caused by: java.lang.ClassNotFoundException: org.springframework.roo.bootstrap.B
ootstrap
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
Could not find the main class: org.springframework.roo.bootstrap.Bootstrap.  Pro
gram will exit.

The easiest work around is just to make sure you create them on the same drive, and then it works fine.

I’ll update more as I get chance to play

BTW – M1 also includes Selenium support

Quick update:  this will now be fixed in the next release JIRA task here

The easy andwer is everyone should jump and fix it as soon as possible, not later, not tomorrow, NOW!

OK, so you find what’s wrong, fix it, create any tests you need to make sure it soesn’t happen again, and then you Bring on the Coat of Shame!

Development should never be a blame game, but a bit of humiliation never hurt anyone…If you check in code that breaks the build you get to wear the coat.

Now if we were in a blame culture we’d make you go to the pub wearing it and buy everyone a round of drinks wouldn’t we?

The Coat of Shame!

And the coat? No, not mine but it really does belong to someone where I’m working now!

Doug Lea – http://gee.cs.oswego.edu/dl/
As poor software engineering goals, some criticisms are valid (but it needs to be looked at in comparison to it’s modern peers), though whatever tool your given, people inexperienced in it’s application will misuse it in ways in which the originator never intended. After all, you can still bash a screw in with a hammer!

Is this also a question of how engineering has changed? A colleague recently made the comparison of how we spent childhood playing with tape recorders, taking things apart to see how they worked, and doing basic hardware hacking, and what current 10-15 year olds do and how they approach problems. They seem to be more a generation who seem to be at a loss if they turn something on and it just doesn’t work straight away – many don’t understand how to take a step back and systematically debug what is wrong. Perhaps they’ve been spoilt by the Internet, and mobile phones – the way it just works. They have no idea of the underlying protocols and the what or why they do things so have trouble reasoning based on small building blocks of information.

To quote Blackadder -

E:  Yes. He has patented a machine called "The Ravelling Nancy".

PR: Mmm, what does it do?

E:  It ravels cotton sir.

PR: What for?

E:  That I cannot say sir. I am one of these people who are quite happy
    to wear cotton, but have no idea how it works.

• Download the xpi file, this is easiest in a browser other than Firefox (yes sorry to say that!)
• open xpi with tar or winzip
• Edit install.rdf
• Change version number line e.g. 3.0.0.* to 3.1.*
• zip it back up
• Open file in Firefox to reinstall
• restart Firefox

WARNING/Disclaimer: This is an approach for the brave or those who are desperate to get their Greasemonkey extensions working again.

It’s by no means scientific, but I use Greasemonkey a lot (and a couple of other extensions too), and really wanted it back!

What I have: Firefox 3.0 RC1, Greasemonkey 0.7.20080121.0

1. Go get yourself the Nightly Tester tools (2.02) here
2. Make a note of all the extensions/add-ons currently enabled – consider this your safe list!
3. Install and enable it
4. It will ask if you want to override the extensions and enable them – say yes.  Note: at the next step Firefox will almost certainly fail to load!
5. Let Firefox restart – it may well fail!
6. open a command window and cd into the Firefox install location.
7. Run ‘firefox -safe-mode’ to restart in safe mode
8. Click the disable all extensions and continue
9. Once restarted, go into Tools, Add-ons and enable all the things in your safe list.
10. Enable Grease Monkey
11. Restart firefox
12. Done!

Well…hopefully.  If not restart in safe-mode again and disable greasmonkey and check all else is still ok.

If you have other extensions you want to get working, you can repeat this process.  I did manage to get a couple of others working, but not Google Gears

Exploring the OASIS reference model, so that we can have a reference point for understanding and articulating SOA.
It’s very tough when you ask questions like:

- how does SOA differ from other types of architecture?
- If SOA is x, what is the antipattern?
- How does Java relate to SOA?

We need a common point of origin to let us have a sensible discussion about SOA because we all say it is different things, and this is not aided by the product vendors.

So, what is the RM?
- An ABSTRACT model for comparison and analysis of a range of Service Oriented architectures
- A framework for understanding relationships among the entities in a SOA environment.
- Industry standard (OASIS standard-2006)

RM is obviously a reference model, but also by not tying itself to existing technologies it is durable.

The elements are a Service, Visibility, Service Description, Interaction, Contract and Policy, Real world effect and an Execution Context.
The RM is guided by patterns, reference architecture, related models and concrete architectures. It considers protocols, profiles, specifications and standards.

Service: A mechanism that brings together needs and capabilities.
Service Description: declaration of aspects required to interact with the service.
Capability: a specific set of functions resulting in a real world effect.
Visibility: How those with needs and capability see each other and interact.
Execution Context: technical/business elements that allow information to be exchanged, actions to be performed and enforce policies and contracts
Policy: constraints imposed upon invocation.
Real World Effect: The result of an interaction with a service.
Interaction: the models for using the service.

So, the RM covers all the things you need to successfully produce and consume a service.  It also helps describe the critical success factors e.g. manageability, trust, predictability, real world effect, governance, that allow us to communicate successfully.

It’s 2008 it must be the year of Complex Event Processing because BPEL is SO last year

SOA and Java – Differences
J2EE has a single domain of ownership – SOA has many
Java has reliable semantics (some)
SOA is more about the interfaces and the effects than the implementation
SOA needs more thought and design to establish those formalisms

SOA and Java – Similarities
A Service and a Java API are notionally similar.
Encapsulation, Service Transparency/Autonomy are similar
SCA and SDO are good SOA implementation approaches

Some good tips on getting SOA done in Java:

• Think in terms of services, and organise your teams around them
• Think of process and the UI second
• Think about measurements for the service – business SLA
• Think about standardisation. How are interfaces defined, versioned, verified and how are dependencies managed?
• Pick the appropriate technologies and standards – e.g. JAX-WS, SCA/SDO, Apache, IBM etc.
• Maintain consistency in the model – remember that that is what the RM brings to Java
• Remember chang is likely insulate yourself with proxies
• REST can also be compatible with the OASIS RM.  If you know Steve, you know he LOVES REST

Then there was a good live demo, pulling latest builds of vendors RM’s and making it work, taking WSDL into Flex, which uses Apache Axis 2 to generate stuff.
This all works because of a reference model and because standards are a point of reference for vendor implementation.

The first release candidate for Firefox has just surfaced.  You can get it over at Mozilla.

It has a nice new look, a really useful address bar, and it is so much faster than 2.

A talk around implementing the OpenSocial APIs for your site.  OpencSocial defines a dingle API for social applications that covers the majority of interactions with social web tools.

There are two basic parts.  The Server side contains a Gadget Server (think Google gadgets), and an OpenSocial JavaScript library. The client side uses XML and JavaScript, similar to that by Google gadgets to make requests to the server using JSON requests.

The JavaScript API has 3 constituent parts:

• People and Friends API
• Activities Data API
• Persistence Data API

So what are your options? Well, you can either roll your own API to the spec. or utilise the existing open source implementation – e.g. Shindig.

An opensource implementation, as part of the Apache Incubator.  Currently only for PHP and Java, and very easy to use.

The Gadget server renders XML, and the server-side components are covered by an OpenSocial Servlet, and now there is a REST based API on it’s way.  Note: It needs JDK 1.5.

It comes bundled with Jetty so you don’t have to have another Application server to run it in. When you download it, there is an infrastructure dir with some useful plugins.  A maven plugin for Eclipse that allows you to great a project for a gadget etc.

When it comes to integrating it with your data e.g. a Database, you just need to implement your own brand of the sample implementations included of People Services, Activity and Data.  These are bound using a provided module, using Guice.

Security is obviously important, so you can either use the built in OAuth mechanism, or again roll your own.

Social site Hi5 have built their own, based on the RI, more detail on their developer site.

They shared a few lessons learned:

1. Prepare for viral growth.  They experienced a sudden 50% increase in days.
2. Think carefully about the policies – what you will allow
3. Build tools to for publishing and managing this data.
4. Think about the request model – they had a friends list on every page, which each time the page gets refreshed, makes a very expensive call to getListFriends.  They added a TTL style cache.
5. Tuning – on their dev site, there’s a tuning guide for 6.10
6. There is a need for lots of synchronization and encryption (cases that worked in the guide)

This session was really about how you can be more focussed in software development by utilising tools and process to take out some of the pain.

Main focus of which was:

• Reducing bugs
• Using build scripts
• Managing dependencies
• Automating builds
• Checking code quality
• Introducing tighter integration with issue tracking
• Automating the generation of some technical documents

Build Scripts: The aim is to have something reproducible, that works anywhere (not just on my pc!) and is portable. Once you have this basis, then you can look at automation through Ant or Maven 2.

The two main choices (though there a few hybrid ones now based on some of the dynamic scripting languages) are Ant and Maven 2. Both have advantages and disadvantage, which is why Apache maintain two!

Ant is widely used and extensible, but does need a lot of low level scripting, and doesn’t have a standard lifecycle (there are conventions, but they are not consistent).

Maven is a declarative build scripting framework that prefers convention over configuration. By following the conventions yourself, much of the ‘lifting’ becomes simpler.

Both support lots of plugins/tasks for deployment, reporting and quality measurement, but because Maven has a standard lifecycle, unlike the adhoc nature of tasks in Ant, it is much easier to understand what is going on in the build process without having to wade through the build source scripts.

• Standard directory structure
• Convention over configuration
• Better dependency management
• Good support for multiple concurrent projects and the dependencies between them

Although Maven imposes a standard directory structure, which yes, isn’t as flexible, but it does mean as you go from project to project you don’t have to work out each time what to put where.

Managing Dependencies: Both internal and external dependencies, e.g. libraries like Commons logging. Traditionally, each project would have it’s own JAR files, but it is hard to keep a track of versions of common JARs you are using (and the ones that are in turn used by things you depend on! Which may well differ). So to get around this we can use Ivy for ANT, or and ANT task for Maven. It’s also worth maintaining a local repository of libraries you depend on, versioned in some sensible (even if it’s renaming them with the providers versioning) way so you know what versions they are.

Good testing practices: Remember that Unit tests document your intentions! they also give comfort – not just to you. More readable cases are easier to maintain and regression test.

The two main alternatives are TestNG and JUnit

TestNG is certainly increasing in popularity, and it looks like JUnit 4.4 is making testing much easier over JUnit 3, with:

• Annotation based testing
• Ability to test for timeouts
• Easier Exception Handling
• Parametrized tests

It was certainly a pain before that everything had to be derived from TestCase. Now any class can be used via annotation @Test. Before there was setup and tear-down, now there are @before, @after, @beforeclass and @afterclass. BTW – JUnit 4 still runs JUnit 3 tests.

Now that it is any plain Java class, and the fact that methods can have any name, developers are encouraged to use behaviour driven testing, for example naming the test method ’shouldUpdateEmployee’.

The parametrized testing is a significant addition. Basically, it takes a set of arbitrary data and test cases in a single bundle. You just need a test class with fields that match the grid of your test data, and some data as rows – it then executes once per row, which is a great improvement – you can now test more data with less!

They’ve also added Hamcrest asserts (see the tutorial here) a much more readable way of using assertions. So instead of the rather weird looking assertEquals(10,x) you can have something like this:

assertThat(theBiscuit, is(equalTo(myBiscuit)));
or
assertThat(“chocolate chips”, theBiscuit.getChocolateChipCount(), greaterThan(10));

It also generates much more friendly error messages e.g. ‘expected red, got blue’.

Test Coverage: The aim of which is really to help you write better unit tests! Obviously, there’s no guarantees that what you have is well written, but points you at areas of your app that are poorly covered.

Add it into you build process to give early feedback and visibility to the team (see Cobertura which is great for project level coverage reporting), or build it into your IDE (tools like EclEmma).

Another useful tool I’d not seen before was Crap4J – boy do they need to work on the name! It gives some good additional statistical data like the number of times it was actually executed too. Sounds like a good addition to the toolkit of any build-meister.

One good point that comes across again and again with many of these tools – not just for test coverage is Why use one? Use both! Chances are they’ll pick up different things.
Continuous Integration: Basically, automatic integration testing and compilation on a central server. Obviously better with some sound testing, and good coverage so I’d sort that first!

You need an automated build process (which you should have already with ANT, maven or similar), automated test processes and a source repository.

Common tools – Continuum, Hudson or Bamboo. I’m going to a session on Friday by Kohsuke Kawaguchi, Hudson’s creator, which should be cool – more later.

So what problems do they solve?

• Infrequent Integration and testing
• Infrequent commits – because people are scared of breaking things, which is ironic really
• Fewer release, which results in less user testing
• Reduces the number of bugs over the lifetime

So, no excuses.

Other good stuff you can leverage from your builds – take a look at the build history. What you want to see is an increasing number of tests, and some indication of build stability e.g. 5 out of the last 6 builds were all successful. Go one step further – auto update your wiki to show it, it gets people more involved in the build process, and fixing it if there are problems, but it also shows when your heading in the right direction. See testing doesn’t always have to be negative

User perception: What else could you do in your build process? How about automatically deploying a successful build occasionally, so that users have got something to see, and can see some progress. Try and show them some way what’s new, so they can see the fruits of all your hard work. Continuous Integration and Continuous feedback (hopefully), not bad.

Code Quality: Increasing code quality give you:

• a better product
• a more maintainable product
• a descriptive statement of intents
• enforcement of coding standards
• an opportunity for training and development of new team members. Plus possibly those who by now need a refresher course!

Usual suspects CheckStyle, PMD, FIndBugs and Crap4J.

CheckStyle gives good pointers on coding style and formatting. FindBugs is good at tracking down potential bugs i.e. those that may bite you in the future. PMD, a bit like CheckStyle, but more technical. Crap4J also gives good indicators of poorly tested classes and overly complex ones.

Automated Documentation: What??? I hear you cry. No, not fully automated, but how about some basics that you should be doing that you can at least part generate automatically, and will always be up to date? Typically wjhat happens is some of the basics are written once and then forgotten as everyone moves on to the next important thing. They are still basics though.

Obviously they will be a little dry, but two good ones to try : UMLGraph and SchemaSpy

UMLGraph will be clickable and navigable class diagrams right into your Javadoc, which is a great start.

SchemaSpy is similar, but for Databases. Given an arbitrary schema will generate a nice ER model for you. This can often be really useful, especially if someone has given you a large legacy DB, and no documentation, just reverse engineer it.

As ever with useful documentation like Javadoc – generated or not, note that you may need to use thumbscrews to get people to actually read them!

Final tips:

Fail early and often in the build process, and give feedback ASAP. Failure at any stage (unit, integration or quality) halts the build. Tests getting too big? work out which ones give the best indicators and run them always. Then move longer running indicators to a less frequent schedule e.g. at night.

Current trends to reflect the move to lighter technologies. However, the patterns for scaling don’t change, and this was a session aimed at giving guidance to achieve that…without shooting yourself in the foot.

10. Understanding the Problem

• Initial expectations are crucial.
• A scalable system will always be slower in single user mode
• Complexity always increases path length

Indeed, if a system is fast in single user mode but slow under heavy load, then the problem is not one of performance, but scalability.

Horizontal scalability is the only answer

9. Define the requirements

The ARSPMS Requirements model
Availability
Reliability
Scalability
Performance
Manageability
Serviceability

1. Understand the theoretical achievable performance – define single user load scenario
2. Understand the cost of achieving these capabilities.
3. Work towards ensuring linear system scalability, i.e. making sure the cost per user doesn’t rise (what you see when adding more servers, each successive addition supports less users than before)

Gotcha – Predictable latency
more moving pieces
- Garbage Collection
- Thread switches
- Network I/O
- Switches and Routers

Gotcha – Read Consistency
2nd law of distributed systems – The only things you can distribute are state (info) and behaviour. This presents two challenges
- distribution creates multiple copies – how do you keep them in sync?
- how do you guarantee transactional consistency

Gotcha – Durability
Achieving availability of stateful system requires a form of durability e.g.
- sync replication
- disks/SANs
- transaction logging
- durable message queues

The challenges then are:

• latency
• shared access between writer and any potential failover

8. Architecture trumps technology!

I like this one, and it’s at the heart of many of the others here i.e. you can’t just throw more technology at the problem. There may be natural domain specific limitations, sequential high latency operations, globally ordered operations or data hot spots.

On the flip side, data consistency and reliability, immediate reliable state access, durability of state change.

Red herrings – in truth, these are all scalable…
EJBs, tiered architectures, RPC, XML, Web Services, SOAP, HTTP, Programming languages and Operating systems. They may well be inefficient but not un-scalable!

There are some legitimate language and platform concerns that should worry us:

• predictability of latency – unpredictable scheduling and duration of full gc pauses
• Lack of control over thread scheduling
• Lack of asynch I/O
• Ease of development – languages differ in effectiveness for special tasks
• Availability of libraries and components, after all we don’t want to keep re-inventing the wheel..especially if someone smart like Doug Lea wrote it

7. Understand the basics

a) Delivery – packet oriented, stream oriented, req/resp
UDP /IP Packet oriented – not reliabile, limited payload size but predictable latency

TCP/IP – stream oriented, unpredictable latency but easy programming

HTTP – req/response based, built on top of TCP/IP

b) Messaging concepts
delivery guarantees, latency, processing model (at best once, at most once etc.) and ordering guarantees

Efficient models are single threaded, reliable, in order, zero latency once and only once – like being sat at your workstation.

- messaging makes you pay

• ordering limits scalability
• reliability adds costs
• introduces latency
• once and only once is hard

6. visualize the network
The wire – a few basics e.g. Ethernet and Infiniband, all very fragile

Switched fabric offers fault tolerance, scalability and direct p2p connections.

5. visualise the design
Patterns of stateful scaleout

Sprayer – no knowledge of app
Routing – to the right server
Patitioning – assign responsibility
Availabilty
Messaging
Coordination

Routing – information and processing in right place where there is locality of state. Reliable routing is fundamental and supports partitioning and replication.

Partitioning – ownership of state distributed

Replication (availability) – limit loss, relies on coordination

Coordination – difficult to scale and can work against availability, orchestrated state change process.
benefits – simplifies programming, supports synchronous behaviour

Messaging
Queue supports reliability(hand off), durability, ordering, collection (batching – a bus load). He made a good point about messaging, in that if you have enough localised queues, when you go to get a message from the queue, it’s not always best just to get one – think about batching work, why not take a bus load?

4a plan for overload
load balancing – spraying
partition
queue – collect then process in bulk
parellelize – spread concurrent processing.

Oveloaded? What are your options?
a) turn away request
b) Queue
c) add capacity dynamically
d) increase batch size or q delay – let them stand in larger groups
e) relax read and / or write consistency –> sometimes you have to e.g. caching

4b partition for scalability
It’s the only way to provide significant scale. And dynamic partitioning supports dynamic capacity. This introduces a challenge for distributed read consistency. Even with custom affinity of data, complete data locality is simply impossible for some apps.

3a plan for failure
we cause accidents…
The 3 R’s

• redundancy
• reserve
• recoverability

Plan for failure at two levels – component, and system.
When possible failover to transparently handle component failure – e.g. server. Use recoverability to handle everything else (requires durability – tx logs, db logs etc. to reconstruct after failure).

3b replicate for availability

Do
synchronous replication for durability
replicate at levels that are responsible for failover – not everywhere
asynch replication for increased locality e.g. caching or for recovery

Don’t
replicate synchronous inside a critical session unless you have to
Only replicate only what you need!

In most successful models events are partitioned and THEN replicated.

2. tier where it makes sense
- this is not horizontal scalability, just segmenting responsibility.

Pro’s
it can increase scale
localisation can eliminate need for coordination, and thus help to scale
tier traversal creates opportunities for batching queues

Cons
added complexity
potential for reduced efficiency
- tier traversal should be on a constant order

There is no reason to tier stateless systems for scalability! Though maybe for other reasons:
- latency
- scaling
- availability
- efficiency

1. Simplify
Complexity is the enemy. Reliable distributed systems must be modelled as finite state architectures – prove them
Partitioning comes with hidden treasures – localized ordering managed locally
Consider change to the programming model e.g. use an event driven architecture to turn the problem on it’s head.

Conclusion
if you can’t show it working on a white board…it’ll never work in production. Don’t we know it!