Make Things Simple but not simpler

Gradle archetype for Spring applications

I am releasing a Gradle archetype useful for creating Java/Groovy applications based on Springframework. Of course, it is not a real archetype because such creation it is not possibile, but with very few steps you can create, edit and deploy an application server. It would be the most accomodating starting point for deployable software projects.

This release is an attempt to mitigate common issues related to development life-cycle phases such as testing, the running of application and deployment in various environments. The archetype leverages on the flexible building process and on the top-most featured IoC (Inversion of Control) management system.

When creating application modules for linking services through HTTP, JMS or any other connector type, this archetype is refined and can be applied for satisfying those requirements:

Automatic testing, building and continuous integration.
A Different configuration for each environment (development, integration, production).
Springframework based system.
Groovy support.

The project consists of:

Utility Classes for given Spring context.
Grails-like DSL for Spring setup (beans.groovy).
Logging and application configuration properties for each environment (development/integration/production).
Gradle config file.

Why Gradle?

Problems exist using Maven in Groovy projects due to the gmaven plugin, which may indicate that it is not ready for the groovy-user community. Indeed, Gradle works perfectly on Groovy projects. It is so concise and elastic that you don’t have just a building system, you have a programming tool. When a customized behaviours proper plugin cannot be found in the registry, you may add custom tasks by writing groovy code directly to the build.gradle descriptor. Gradle is a swiss army knife for developers.

Getting started

Run ‘git clone git@github.com:gfrison/proto-app.git myApp’ where myApp is the name of your project.
Edit property ‘projectName’ in ‘build.gradle’ with project name.
Add classes, and manage them with spring ‘beans.groovy’.
You are now ready to test, run, deploy your project through a continuous integration system such as Jenkins.

If you have suggestions, or pull requests from Github, myself the author, would be happy to consider them.

Software architect mistakes

I think that to get up in the morning and brew a good cup of coffee is one of the best way to start the day. You know, the heady fragrance that emanates from the machine-pot, it’s delicious. When it’s ready, pour the coffee into a cup, then add some sugar, and finally you got it.
End of the coffee making process.

Have you ever thought to design a coffee making process with some diagrams, or doing the same with other banal activities as taking a shower? Of course, not.
For other cases less trivial than these, including software project development, a minimal-design work can be quite useful and somewhat needed.
Often question arises, is an architecture design worth the time and effort invested in it? Well, you may answer this question first: Are there risks in the project that could be minimized by an early design activity?

The more ambitious and challenging the project is, the higher the number of ‘risks’, and the more difficult it is to complete successfully.

How to identifying risks. The easiest place to start is with requirements, in whatever form they take, and to look for things that seem difficult to achieve.
Gathering requirements is fundamental for deciding what to do and how. However, sometimes problems arise at this starting point that lead to the ruination of the project. Some assumptions may underestimate this key phase and shake the architect role to its foundations:

It’s someone else’ responsibility to do requirements.

Domains drive the architecture choices, not vice-versa. Requirements can create architecture problems. At the very least, you need to assist the business analysts.

I learn the domain as I write the code; incrementally.

While prototyping pieces of software is a way for mitigating engineering risks and figuring out the hardest problems, writing code could be a waste of time for analyzing a domain. Rather, it’s very cost-effective to modelling it in advance.

The requirements are already fully understood by the stakeholders.

Clear communication is critical between people and the role of a software architect can be a very difficult one when others don’t understand what you do and why.

Domains are irrelevant to architecture choice.

Developers may copy an architecture from a past project. Maybe just following the company standard, but ignoring the motivations behind previous choices. They are more likely to be unaware of the qualities required in the current project.

I already know the requirements.

At least the documentation should be in your mind, but designers should use models to amplifying their reasoning abilities and unfold not clearly visible aspects that affect their own risks.

Cool DSL with Groovy

Domain model design has never been confused with ‘ease’. From the dawn of its conception, generating executable Unified Modeling Language (UML) diagrams meant sweat and frustration. Generating stubs & skeletons, alone, led one into quagmires of Java Enterprise Edition. Yet it is inherently possible — and actually has been for ages — to design a programming language dedicated to solve specific domain problems; effortlessly and quickly.

DSL is not a foreign thing. Developers are steeped in DSL, even if unwittingly. Frameworks are DSL. A macro is also DSL.

When you write a function? That too is steeped in DSL. But functions are often a dirty business; un-standarized; quirky, whereby even domain experts face a daunting task when unraveling and then being put to task to update such rapscallion boilerplates (that is if they do not themselves compound the problems).

A programmer worth his reputation would find it necessary to isolate and clean up all non specific domain terms within the DSL. This literally means rooting through and setting aside all secondary-in-importance setup code, allowing the domain expert to establish or reestablish the primary foundation code.

This is most times done for large money for an end user; the helpless customer.

Given an appropriate DSL that fits their needs, customers could write all of the
code that they need themselves, without having to be programmers.

Now let us imagine, or better, take for granted, that a good DSL experience can exists.

That given an appropriate and standardized DSL environment, even the customer,

with a few simple tools and grasp-of-concept, can self-write code, gracefully, and

specific to their needs without the cost and hassle of having to become, or running to,

programmers. And that is where Groovy comes in.

What’s Groovy? It’s Java as it should be. Easy and intuitive, it offers new features unknown to its parent yet (I’m still waiting for release 7), and come up with those benefits that form the basis of the DSLs that we will develop.

A fundamental feature is the MOP, the ability of changing runtime the properties and the behaviour of objects. It allows us to respond to method calls that do not exist in the class, in other word to “pretend” that these methods exist.
Another essence to consider is the Closure. It’s the real power of Groovy. The extreme flexibility of this kind of object/method allows us to change its behaviour replacing its delegate class, on the fly.

Closure delegate. Groovy has three variables inside each closure for defining different classes in its scope: this, owner, and delegate.

The this variable refers to the enclosing class. The owner variable is the enclosing object of the closure. The delegate variable is the same as the owner, unless that delegate is substituted.

class MyClass {
  def closure = {
    println "this class:"+this.class.name
    println "delegate class:"+delegate.class.name
    def nested = {
      println "owner nested class:"+owner.class.name
    }
    nested()
  }
}

def tryme = new MyClass().closure
tryme.delegate = this
tryme()
/*
output:
 this class:MyClass
 delegate class:Script1
 owner nested class:MyClass$_closure1  
*/

When a closure encounters a method call that it cannot handle itself, it automatically relays the invocation to its owner object. If this fails, it relays the invocation to its delegate. one of the reasons builders work the way they do is because they are able to assign the delegate of a closure to themselves.

Walzing with the Tech crunch

The economic crisis we’re currently going through is teaching some lessons to the Western countries, in particular to the Anglo-Saxons, that our grandparents know pretty much, although it seems we’ve forgotten the past years in this financial bubble. The debt has several pros: allows building, buying, investing and, when properly managed, might ensure a safe return and a fair growth of the economy. However, the debt has an outstanding bad side: it must be paid back. It might be postponed, rolled over, shifted to other (more or less conscious) subjects, and its dreadful effects would be identified as bankrupt, credit-crunch, real estate bubble and recession.

With a view to the software applications, a similar observation might be rightful in terms of ‘state of health’, which point to a family of properties that the software should have to be easy changeable, so it could respond quickly to the requirements evolution. A software that doesn’t enjoy good health is the one that has become fossilised to the original architecture, keeping it as is as possible, never revisited in the light of technological innovations and functional updates, but just patched with improvised and unconvincing surgery.

It’s suffering what it could be defined as inability to bear the debt built up over time, but in this case we’re not dealing with financial debt, this is the technical debt. Even though the term ‘technical debt’ sounds strange, it’s related to the financial fellow in many ways, and it is widespread in software development. The saying according to which economy is based on credit (debt) finds support also in the software world.

Developers who are reading know well what I am talking about. You’re assigned to work in such XYZ firm from next Monday for at least 3 months, and when you’ll start this new task you’ll be instructed about what to do.

The workout mainly consists of implementing new features on top of the customer’s solid rock application, a very remarkable system built some years ago for serving peculiar needs.

So far, it has worked well, the owners said, you may just make it worse than it is now. Later on, you have no choice but to agree with them.

Expanding or changing the set of features without re-factoring looks like seeding a crop without ploughing the land before, if the system’s authors didn’t predict such an amendment. The harvest could be lost, couldn’t it?

You’ll be asked to complete your job updating the old system and keeping the structure as it is, avoiding to break the fragile balance among components.

Just for you information, consultants were called few months ago for a similar task. They added such a mess into the code that you have to spend most of your time to figure out what they wanted to do than working effectively on new things. Maybe the customer were disappointed by their way to conduct the development and now it’s your turn.

Thus, in addition to the new enhancements, you should fix what your precursors did.

This subject is hard to handle and quite unpleasant in particular when the customer doesn’t want to hear talking about re-factoring unless it doesn’t delay the delivery, which is almost impossible, so the scheduled task proceeds as expected.

In short, it looks like going for a walk over the broken glasses swearing you won’t be injured.

Using the post’s subject, it looks like getting into debt again for covering the old one, just for adding short term solutions when too many of these have been applied in the past.

Looking back at the past, I’m realising how this kind of intervention is predominant on the amount of works done, that I don’t know how much time I would have to wait unemployed if I wanted to work only into brand new projects…

Sometimes I’d define myself as a debt collector, and I find it uncomfortable as a lawyer or a doctor would feel against a criminal prosecution or a rescue surgery: it’s an exploitation of other’s misfortunes. It might be painful, but we make the customer feel better.

Powered by Apache Mina

The application being discussed has to behave in the following way: perform the client authentication, accomplish the request and response operations and forward notifications asynchronously to the client. The Mina framework fulfills these needs because it was created to be as much flexible and easy-fitting as possible to the most of the use cases. Mina is structured in several layers, that briefly they could be listed on those which concerning about: input parsing, execution of concatenated processes and serialisation of the related responses, if needed. The infrastructure take care of the I/O tricky affairs, while it lets you write upon it your business processes and handle the session lifecycles through simple callbacks, that you can manage within few codes. Majestic! I love it.

Furthermore, I was looking for something that could help me to write down the application which implements many commands, preferably in a good looking way, where each piece of service could be isolated from the rest of the application. Even this fancy of mine has found satisfaction at once.

The demultiplexer is a device with a single input and many outputs. Its role is to select the output line accordingly to the context rules. This pattern is also implemented in Apache Mina for writing decoders, handlers and encoders and I found it onto demux packages, they are: DemuxingIoHandler, DemuxingProtocolDecoder and theDemuxingProtocolEncoder.

Apache mina architecture
Filters
They could be used for several purposes: I/O logging, performance tracking, thread pooling, overload control, blacklists and so on. In my case I configured two filters, one for the user authentication, and the other for thread pooling. Once the user is logged in, the authentication filter removes itself from the client session filter list, while adding the one which transforms the raw input into POJOs.
Decoder
The TCP server must implement a proprietary protocol, it’s a simple ASCII protocol for exchange commands, their length is undefined and the end point is a sequence of termination characters (likewise SMTP). Therefore, I extended the CumulativeProtocolDecoder class that gathers the input until the end of command, and leaves to us the parsing of the bytes and the creation of a simple Java Bean. After that, the bean is going to be driven to the filter chain to be executed.
Handler
One of the IoHandler implementations drew my attention while I was looking for something resembling the Command Pattern. Each message coming from clients means a specific action, and I find it so tedious writing a single Handler that switches operations by the type of the incoming request. An elegant solution is provided by DemuxingIoHandler that posts the requests toward the corresponding handler. The handlers have to implement the MessageHandler<E>, the generic type defined will be the object’s class that the DemuxingIoHandler will submit to the handler, and register themselves invoking addReceivedMessageHandler(Class<E> type, MessageHandler<? super E> handler).

The asynchronous nature of Mina allows to handle a huge number of clients by an handful set of threads. A further decoupling between I/O and business logic may be done by the ExecutorFilter, which is in charge of the messages after the NioProcessor.
Encoder
The transformation component works in a reverse way compared to the decoder: it serialises the POJO response coming from the handler process, to the output stream, toward the client. Likewise the handlers, it is possible to delegate its own encoder for each response object, but why not to send the IoBuffer straightly from the handler that elaborates the request? Separation of concerns might be the answer. The handler receives a command, a java bean, then it is processed and the handler returns an object for response, a POJO. It’s up to the encoder to transform the abstract response to a concrete message through the agreed TCP protocol.

Recruiter advisory: Explicit lyrics

“I’m a people person, very personable. I absolutely insist on enjoying life. Not so task-oriented. Not a work horse. If you’re looking for a Clydesdale I’m probably not your man. Like I don’t live to work, it’s more the other way around. I work to live. Incidentally, what’s your policy on Columbus Day?”

You, Me and Dupree (2006)

This is the interview every recruiter would want at 17.00 on Fridays, so fast to let you step out soon for the forthcoming weekend, plain and clear in the outcome.

Usually it’s not so an easy job for the Head Hunter, selecting people and finding the right ones to slot into the pending position might be hard. The challenge would be more difficult when they are seeking to recruit through controversial methods, which hardly could achieve the wished result. As human beings, we have the natural tendency to think that our choices are rational, while we underestimate the effects of the undercurrents that, in a way or the other, affect our decisions. We believe to be steady inside the boat in the middle of the sea, even if we are at the mercy of the weaves. We are prone to get swayed.

Fortunately, the good recruiter studies books deeply and prepare himself in the training workshops to get rid of those diversions, and then finally he can apply scientific methods to his job. Progresses in this field could be checked when they act like CIA agents asking questions as “What will you do when you grow up?”, “When was the last time you were happy?” and again “What are your strengths and weaknesses?” rather than “Tell me about yourself, describe yourself in one word”. But, dear recruiter, I can’t describe myself in one word, unless it’s both hyphenated and a metaphor.

What comes first, if I’m talking with the interviewer for the job, is that I want to check if his expectations match with mines. I’m talking with you to show my professional skills, not to talk about my passtime hobbies, neither about dance nor motocycles.

All those requests never end to astonish me for their futility, even if those make sense just for HR department, I’m not going to dig into the matter.

Behaviour interview

A much more effective approach is to conduct very structured interviews where the questions are focused on experience, skills and ability rather than vague things.

What recruiters sometimes try to follow is the behaviour interview assertion, which declares that the most accurate predictor of future performance is past performance in similar situations. It would be enough to fright any financial mentor but it finds logical basis for canditates’ evaluation. Perhaps the behaviour of a single man is easier predictable than a stock index. HR specialists claim that with this method of leading interviews, it’s much more difficult to get responses that are untrue to the candidate character, because these should be detailed descriptions of past events, or experiences faced at work.

I agree on the idea that past experiences are indicative on how people react under certain circumstances, but I’d put less emphasis on that, first of all because challenges are always different. Whatever technological issue the company is facing right now, it would be far from any candidate experience, recruiter may figure out something else by the applicant. What someone has done shows the ability to execute; personality is important, intelligence naturally more so, but improvisation remains the key. Knowledge workers must adapt their knowledge to the situation, but if during the interview the cadidate isn’t projected on a real scenario to show his capabilities and past learned lessons, how the recruiter actually could form an objective opinion?

I was rarely asked advices or opinions about real technological matters involved on development, is that the reason the recruiter doesn’t know much about what the new employee is going to resolve?

Maybe sometimes inteviews are not used for hiring people, but just to gather information on cadidates, to create statistics on salaries, skills and to estimate how long does it take to search for a special kind of professional in the market, I guess.

Money

Recruiters may discard people based on salary, of course they can; especially if the point is that the people are interchangeable, low cost and easily replaceable like a natural resource. usually this happen when the target cadidate is junior.

Salary is a complex issue the more senior the target is. Seniors want to discuss the context of the job before they ask about money. Answering the salary question in a phone screen or in an interview before building rapport drop me to the disappointent, as the recruiter is telling me “We want the cheapest on your position”. That’s ok, but do you want to save money before you know what I have to offer? Or, why are you looking for someone senior?

HRs usually match your CV keywords (better know as buzzwords) with their table axis to define your salary box, framing cadidates in a very simple way. Although the salary offer is equals among peers, a fascinanting metric highlights that 5% of programmers are 20x more productive than the other 95%. Now, let me know in your opinion which section of this statistic is firstly discarded.

Quiz

I don’t see anything wrong with the interview questions with multiple choices, sometimes they are as funny as filling in crosswords, but some other times these questions upset me for I realized I forgot some exponential functions since the school… Damn!

Although it would be helpful to filter out applicants without a basic education, in several years of work I never had concerned about exponential calculus to strike a business requirement.

What's the bottom printed row of this function?

for(int i=0; i<30; i++){

   System.out.println("line: "+i);

a) line: 29

b) line: 30

c) none of the above

These requests end up by annoying their prospective employees, any company would lose appeal, dropping any willingness to get hired by the company.

This is a newbie question, then the recruiter answered me telling that every technical employee in the firm had filled such a questionnaire. Really? I don’t think any of the experienced programmers I know would waste time crisscrossing questions like that on a job interview unless they are hopelessly unemployed, and if the hiring manager is looking for an experienced developer, why ask these first-level programming questions? If the recruiter can’t read the resume, why would a hiring manager?

7 steps to MDA revolution

I didn’t believe that such a successful project was such a rare event in the IT industry, that’s why I’ve never caught another chance for applying the learned lessons again. I thought that the experience accrued on Model Driven Architecture will be reusable in other circumstances, though I’ve never seen concepts as executable UML or MDA either applied or mentioned in the following commitments I’ve pursued into.
The idea of this project wasn’t conceived by external consultants thirsting for selling their cool technology; instead, it was born and grew up just inside the development team. The architecture’s transition had been gradual, and little by little, as new automation scenarios penetrated our excited minds, we moved as many as possible development processes under MDA framework.
Despite my early impressions while considering to undertake the project, the upper management embraced it and laid down investments counting on the benefits that this new approach would provide to the development.
What is difficult to change is the modus operandi of a 300 employee company that offers banking services and applications, which is engaged in one of the most conservative field in technology and development methodologies by default. It was about a significant jump in the services development and as the PM remarked:“We are developing as dinosaurs, don’t you know what the hell happened to them?”, the way to MDA was traced.

The issues we faced with the introduction of modelling notions would be defined as practical contingencies rather than theoretic or philosophical reasons, foremost the mess in the business layer. It raised reliance and maintenance weaknesses with time, even security holes that sounded so bad in such a company with a plenty of banks as customers.
The hundreds of cases developed by dozens of engineers turning over throughout the months in the java development area had reached the critical mass, enough to trigger an explosion/implosion of the whole system. On the other hand, the applications can stand up only by high costs of maintenance and lazy deliveries, due to the difficulties on integrating incoming services with the underlying system.
The application layer managed the data flows between clients at the top and feeds and legacy information systems at the bottom. On their way, they affected several mixes by business process rules hardcoded in obscure java classes. Unfortunately, most of those shaking details were lost, because of the policy related to the development, which didn’t claim about missing documentation, and then it was so damned annoying to go back and take over old artefacts for maintenance or updating rules. Only skilful programmers might extricate the balled up code. The critical mass had to drop down and be brought to lower temperatures quickly. New developments and dozens of incoming features were planned, so a deep refactoring was a must; it can wait no more.

How was the domain layer implementation that popped up the highlighted problems?
The developer’s effort was mainly focused on the creation of java classes implementing a Command and defining the service to the framework through an xml descriptor. The input and output of such a command was a raw DOM argument, which was parsed to extract the input data needed by the business transaction, the most part of coding was regarded for parsing and filling the response’s service that was a raw xml document too. I think it isn’t agreeable to put most of the developing efforts merely on managing input/output data and mapping, but this was the daily job.
Apply the MDA take time, it was an one year evolution, and it would be summarized with:

XSD barriers. It was necessary to set some boundaries for developers, in order to get a minimum of control over the data flows. Each service had its own formal validation on input/output data, though no restrictions were settled on how implementing the services. Never ever elements or attributes not defined in advance by commitments.
Pojo. Replace the raw document with simple pojo as an argument in the call-back methods; this operation aggregates the formal validation with an easy approach on data manipulation. The binding xml-java isn’t hurdle, it is automatic and many available libraries can accomplish this step.
First hints with EMF. Xsd files are models for xml data, EMF is a MOF java implementation, a general abstraction for writing all sorts of models, I don’t linger over it now, but it represented a jump to the service modelling. EMF is an open source library enclosed in the Eclipse platform easy to use and customizable, it aims to separate the abstract model from the ground.
Choice of technology. The play with EMF opened new horizons on modelling facilities. Hooked by this methodology to design SOA applications I realized EMF is not enough, the UML (which core principles are inherited from MOF) can fit much better with my purpose to design the object model, define the process flow and the user experience. UML offers diagrams that you can join together, static and dynamic model may describe most of application structure and behaviour.
Executable UML. What do you do with this bunch of diagrams if you can’t transform them in real artefacts and plug-in them in your SOA framework? Not so much, keeping UML diagrams without related transformations and executions is merely fine for documentation, not much more than this. At that time the company joined the Rational beta-program and I started to develop Eclipse compliant plugins which leveraged the power of UML2 eclipse implementation.
How to define data mapping? One of the main obstacles encountered was the data mapping between two different structures. It happens when you need to connect two or more components inside a service call, and each of those have different data structures. In this case UML doesn’t provide any help and you have to customize the model with special stereotypes and profiles.
Sequence and state diagrams. Class diagram were used to generate java classes, xsd files, copy cobol. Sequence diagrams on the other hand describe the flow of processes and their business rule, even conditional instructions which may be transformed to bpel or custom service descriptors. State diagram shows its benefits modelling the user experience and the steps to complete an operation, it easily tracks the state of sessions and will be transformed into the MVC system, as well as in whatever rich client forms.

Let’s get Scrum

scrum At school times the teacher used to quote an important saying:
culture is what remains after we forget the things we studied thorougly.
The concept is charming, but at that time the principle was often adopted to forget things even before they were studied.

The saying is also valid for software development methodologies, where the best practices try to teach us the right path to come up with something really good, shaping a product in the most efficient way and with the highest quality.
The agile methodologies set few general rules, but the result depends on you, your skills and your team of course, not on the methodology.
Scrum doesn’t produce good software products, but if you are smart, it might suggest you some hints helpful to get away with the failure scenario.

What does Scrum say?

It declares that all activities are in a time box and assigns to each team member his own responsibility based on workload estimation, and the activities priority has to be shared with your chief, most of the time the Scrum master.
The daily meetings are essential, the team members explain what they did the day before and what they are going to do today and the blocking problems that affect the task development, as well as the estimated time for the new task or its progress update.
As in the rugby game from which Scrum took its name, the goal is to get things done. The powerpoint presentations, the docs are internal artifacts but the objective is to get product shipped.

How? By setting objectives for the next iteration (sprint), and incrementally so on with the next. The iterations firstly face the most criticalissues and the trivial ones come later, as you are mostly concerned with the software/system architecture and you’d know if such solution overcome the issue, as soon as possible.

As in the rugby game, the project team will be capable of thinking by itself. The coach hasn’t to enforce a defined set of steps to reach those objectives, and as in a real game the team has to learn to handle chaos of requirement changing and with emerging problems (even hoping that Italy will win the next 6 nations).

My considerations

In my scrum practice, I’ve appreciated the time estimation duty for each single task, as discussing with fellows or build a new feature. However, I don’t find it much helpful during the meetings because it’s an information that project managers need to check the process but not useful to the other developing members. The time estimations are closely related to the tasks, so why not to handle them with the issue/bug tracking system, jira for example? You might use it as a time monitor, so the scrum master can automatically obtain all required informations about the development’s progress.
The meeting is an opportunity to get together and to make it clear to others where you are, but most importantly, firstly explain your problems. Sharing difficulties among the members and get proper tips back make the team more integrated and helps to overcome matters quickly.

What impresses of the agile methodology practices is the communication approach, the synchronization of the developers and the feedback on a daily basis. Quick stand up meetings in the morning, before the activities start; maybe better with a coffee.