Monday, December 23, 2013

Exploiting the /proc filesystem

It has been quite a while since I have written my last article. After pondering what to write for a couple of days, I have decided to share a couple of /proc filesystem tricks. Here is a list of real-life scenarios that illustrate how you can exploit this filesystem:

Environment Variables

Let's start with a simple example. You have a Java process that is misbehaving and you would like to make sure that the JAVA_HOME environment variable for the process context is pointing to the correct JDK installed on the system. Just cat the /proc/[pid]/environ file to view the list of environment variables for the process.

File Descriptors

To get the list of file descriptors opened by a process run an ls command for /proc/[pid]/fd. This directory would list the file descriptors as symbolic links to files or sockets with inode numbers. This is a particularly useful trick in case you are dealing with a long strace output and you have a system call with an associated file descriptor. If the file descriptor is a socket, you can gather even more information by grepping for the inode number in /proc/[pid]/net/tcp, /proc/[pid]/net/udp, or /proc/[pid]/net/unix depending on the socket type.

Swap Space

You are dealing with a server that has started running a process consuming large amounts of memory and the server has swapped some data from memory. It is not big deal for Linux if swapping is not continuous; however, it is a problem from a monitoring perspective because Linux is not going to move that data back into memory. One trick to use the swapoff command to force the data back into memory. If there is not enough space on the other hand, you have to force drop the page caches from memory by running "echo 1 > /proc/sys/vm/drop_caches" and then do a swapoff and a subsequent swapon to free up the swap space.

Connection Tracking

If you ever run into "nf_conntrack: table full, dropping packet" errors... You are using iptables and the kernel is hitting a limit on the number of connections that can be tracked. This is controlled by the net.netfilter.nf_conntrack_max system variable which defaults to 65536. You can either try to increase this default value or cat the /proc/net/nf_conntrack file to figure out what type of connection constitutes the majority of tracked connections and update your iptables rules accordingly to not track them - see the raw table with the NOTRACK option in the iptables manual.

Child Threads

You have a MySQL instance running with a very large connection limit and you have a slave I/O thread that is misbehaving. The MySQL instance is lagging behind yet there seems to be no apparent problem. The next idea you come up with is to strace the I/O thread; however, the "show processlist" output is showing connection ids not Linux process ids. Luckily, there is a solution. If you have persistent database connections, stop the I/O thread and start it again. Then, run an "ls -al" in /proc/[pid]/task which will in turn show you a list of child pids with creation dates. Because you have restarted your I/O thread, the most recently created one should be the I/O thread. You can also verify this by comparing the task creation date to the "Time" column for the I/O thread in the "show processlist" output. If you don't have persistent connections, identifying the correct thread is a bit more tricky. Simply restart the I/O and SQL threads at different times, wait for a while, and then identify the correct thread by comparing the "Time" column from the "show processlist" output again.

Process Limits

Every process has to run within certain limits set by the kernel. One of these limits is the number of child processes that can be spawned by a parent process. For example, when we started running into MySQL connection errors, the underlying problem turned out to be the connection limit (each MySQL connection is served by a child thread) being more than what the kernel allowed for. These limits can be different for each process and can be viewed by catting the /proc/[pid]/limits file. And the best part? You can change the limits dynamically by echoing to this file instead of restarting that production MySQL instance.

Friday, November 23, 2012

Using (R)?ex for Fame and Fortune

Overview

(R)?ex, according to its website, is a tool written in Perl for configuration management and software deployment. The only requirement to have is an SSH connection to the remote server(s).

Alternative Uses

(R)?ex is a very capable tool to automate system administration tasks. For example, I have recently built an RPM build script that copies my local code/files to a build server, builds an RPM package, and downloads the finished RPM to my RPM repository on my local box. I have also used it to generate reports. One recent report was required to see how many of our servers were able to access a certain DNS view.

In fact, the use cases are pretty much unlimited. You can use (R)?ex for any process that requires some sort of automation on a remote server. That said, as powerful as it is, (R)?ex still requires some initial setup to become usable. I have included a couple of my tips below.

Tips and Tricks

Code Structure

Rexify.org Wiki recommends using revision control system externals to distribute your modules among your folders that contain your folders. In my case, I wanted to have a single project checked into my GitHub account so I utilized a symbolic link called "lib" in each of the project folders instead.

/rex
.... /lib
........ /Module
............ Module.pm
.... /project_a
........ lib -> ../lib
........ rexfile_1
........ rexfile_2
.... /project_b
........ lib -> ../lib
........ rexfile_3
........ rexfile_4

Naming (R)?ex Files

The online documentation is not clear about the options you have when it comes to naming your (R)?ex files. The default approach is to use the name "Rexfile". This is not really a good implementation as you don't want to have a bunch of files with the same name sitting around on your hard disk. The solution is to name your files appropriately and use the "-f" for the (R)?ex command to execute your scripts:

rex -f script_name

Code Highlighting

If you are using Eclipse like me, you will have problems when dealing with files without any extensions. The solution is to ensure that you have a Perl file with a ".pl" extension and a "use Rex -base;" statement added to the beginning of your scripts to get rid of all the syntax warnings and errors in your IDE.

Dealing with Hard-Coded Passwords

Remember that the (R)?ex file is actually a Perl script. This means that you can utilize the IO::Prompt package to collect SSH user and password information instead of hard-coding your passwords in each script.

use Rex -base;
use IO::Prompt;

my $username = prompt('Enter username: ');
my $password = prompt('Enter password: ', -e => '*');

user qq/$username/;
password qq/$password/;
pass_auth;

Parallelism

Certain tasks may take longer than expected so do not forget to utilize the "parallelism" call for those scripts.

Logging

(R)?ex output from your script can easily be separated from your script output by piping STDOUT to a file.

rex -f myrex.pl > output.log

However, if you wish your output to look like the (R)?ex output, you can utilize the Rex::Logger module instead.

Thursday, October 11, 2012

Networking with Multiple Guests on VirtualBox

IT Support

I showed up to work on my first day a couple of weeks ago and the first thing that I was given was a company Mac... A Mac!

My absolutely brilliant idea was to ask IT support if I was allowed to wipe the hard-drive clean and install my favorite Linux distribution. After giving me the classic thousand-yard stare that every IT dude gives every time you ask for something, he told me that "it was not supported". Long story short, at the end of the first week, I was left with this piece of brushed Aliminum brick running some applications that does not even run on Linux but required by my new company.

Virtualization

Solution? Of course virtualization. Needless to say, this was a complex setup with multiple requirements: * I want to run a Fedora guest for every day use. There is nothing that can match Konsole running on a bleeding edge distribution out there.
* I also need to run a CentOS guest for certain tasks such as building RPMs and other production related tasks.
* My guest operating systems must be able to utilize the VPN connection provided by the host computer.
* My guest OSes have to communicate with each other.
* Services running on my guest OSes need to be accessible from the outside if necessary.
* I keep the Mac just for establishing a VPN connection and for other company related resources like mail.

I am not going to get into the details of setting up a guest OS in this article. I will primarily focus on the networking setup. You should stop reading at this point and have a look at this superb blog article that documents how each networking option work in VirtualBox.

OK, now that you are back and have an idea about how different network setups work in VirtualBox, we can go over the configuration details to satisfy our requirements.

VPN Access

I have a software token installed on the Mac which provides a two factor authentication mechanism to establish a VPN connection. The problem is this is a TUN device. This means that it operates on Layer 3 (Network) packets. This is an important point because we have to select a NAT network adapter in the network settings for our Guest OSes. The other option - a bridged network adapter - breaks the VPN setup; this is is due to packets being processed on Layer 2 (Datalink) bypassing the VPN setup on Layer 3 (Network).

Inter-Guest OS Communication

The NAT setup to maintain VPN access described above comes with a price. VirtualBox assigns the same IP to all NAT adapter guests effectively preventing any inter-guest communication. We have to have a different IP for each of our guests to be able to communicate. Luckily, VirtualBox provides multiple adapters for this purpose.

First, define a host-only network under global Virtual box network settings. Second, enable the DHCP server for the host-only network so your host provides an IP address to each guest automatically. Finally, under guest network settings, select the host-only network adapter under the Adapter 2 tab. Run the ifconfig command on your guest and you shall see two network adapters: one with a NAT setup for VPN access and the other with a unique IP address for communication.

Outside Access for Guest Services

Port forwarding to the rescue. This is the same concept as configuring port-forwarding on a router. The port forwarding configuration section is available under the advanced NAT adapter settings for your guest operating system. See the link above for more information.

Friday, September 14, 2012

Android OAuth2 REST client with Spring for Android

Overview

In this article, I will try to explain how to design an Android OAuth2 client that can interact with a Smyfony2 back-end implementation using Spring for Android and Spring Social.

Before we start, I need to emphasize a couple of points. First of all, I am not an Android expert; I am still learning. Secondly, I like to leverage existing tools so I have chosen Spring Social for Android for this article. It satisfies my requirements for this tutorial but it may not be suitable for every case. Finally, I will not be explaining the server side implementation in this article. Read my previous FOSRestBundle and FOSOAuthServerBundle articles if you need starters.

Raw Data

Let's first look at the raw data that we will fetch from the server. Because I have been using the Conversation and Message examples for a while now, I will stick with the same pattern in this article. Let's say we would like to retrieve a list of conversations by a specific user from a REST endpoint located at /api/users/4f7f79ac7f8b9a000f000001/conversations.json. As you can see in the JSON output below, the data we will receive is a simple list that contains a single conversation with two messages:

[
    {
        "id": "501ee4b27f8b9aa905000000",
        "created_at": "2012-08-05T14:25:06-0700",
        "modified_at": "2012-08-25T00:08:27-0700",
        "replied_id": "503879eb7f8b9aa505000000",
        "replied_at": "2012-08-25T00:08:27-0700",
        "replied_by": "4f7f79ac7f8b9a000f000001",
        "replied_body": "test",
        "messages": [
            {
                "id": "501ee4b27f8b9aa905000001",
                "user_id": "500b3d027f8b9aeb2f000002",
                "body": "Hello Burak!",
                "created_at": "2012-08-05T14:25:06-0700",
                "modified_at": "2012-08-05T14:25:06-0700"
            },
            {
                "id": "501ef8bb7f8b9aa905000005",
                "user_id": "4f7f79ac7f8b9a000f000001",
                "body": "How are you?",
                "created_at": "2012-08-05T15:50:35-0700",
                "modified_at": "2012-08-05T15:50:35-0700"
            }
        ],
        "from_user": {
            "username": "burcu",
            "name_first": "Burcu",
            "name_last": "Seydioglu",
            "created_at": "2012-07-21T16:36:34-0700",
            "modified_at": "2012-08-05T13:22:06-0700",
            "avatar": "http://s3.amazonaws.com/dev.media.acme.com/avatar/50/00/03/02/07/08/09/00/02/00/00/02/500b3d027f8b9aeb2f000002",
            "id": "500b3d027f8b9aeb2f000002"
        },
        "to_user": {
            "username": "buraks78",
            "name_first": "Burak",
            "name_last": "Seydioglu",
            "created_at": "2012-04-06T16:18:04-0700",
            "modified_at": "2012-09-11T21:54:39-0700",
            "avatar": "http://s3.amazonaws.com/dev.media.acme.com/avatar/04/07/79/00/07/08/09/00/00/00/00/01/4f7f79ac7f8b9a000f000001",
            "id": "4f7f79ac7f8b9a000f000001"
        }
    }
]

Code Structure

We will follow a similar package pattern to the spring-social-facebook package and create three packages:

com.acme.social.api
This package will contain our entity classes. In this particular case, we are talking about three entities: Conversation, Message, and User. This same package will also include our template classes containing our API methods.

com.acme.social.api.json
This packge will contain all the necessary classes that handle serialization/de-serialization of our Java entity classes from/to JSON.

com.acme.social.connect
Finally, this package will contain our connection classes that will bind everything together.

Entity Classes

These classes are pretty straight forward so I will just share the code and keep the discussion to a minimum.

User.java

package com.acme.social.api;

import java.util.Date;

import android.os.Parcel;
import android.os.Parcelable;

public class User implements Parcelable {
    
    protected String id;
    protected String username;
    protected String nameFirst;
    protected String nameLast;
    protected String avatar;
    protected Date createdAt;
    protected Date modifiedAt;
    
    public User(String id, String username, String nameFirst, String nameLast, String avatar, Date createdAt, Date modifiedAt) {
        this.id = id;
        this.username = username;
        this.nameFirst = nameFirst;
        this.nameLast = nameLast;
        this.avatar = avatar;
        this.createdAt = createdAt;
        this.modifiedAt = modifiedAt;
    }
    
    public User(String id, String username, String nameFirst, String nameLast, String avatar) {
        this.id = id;
        this.username = username;
        this.nameFirst = nameFirst;
        this.nameLast = nameLast;
        this.avatar = avatar;
    }
    
    public User(Parcel in) {
        readFromParcel(in);
    }

    public String getAvatar() {
        return avatar;
    }

    public Date getCreatedAt() {
        return createdAt;
    }

    public String getId() {
        return id;
    }

    public Date getModifiedAt() {
        return modifiedAt;
    }
    
    public String getName() {
     return nameFirst + " " + nameLast;
    }

    public String getNameFirst() {
        return nameFirst;
    }

    public String getNameLast() {
        return nameLast;
    }

    public String getUsername() {
        return username;
    } 
    
    @Override
 public int describeContents() {
  return 0;
 }

 @Override
 public void writeToParcel(Parcel out, int flags) {
  out.writeString(id);
  out.writeString(username);
  out.writeString(nameFirst);
  out.writeString(nameLast);
  out.writeString(avatar);
  out.writeSerializable(createdAt);
  out.writeSerializable(modifiedAt);
 }
 
 private void readFromParcel(Parcel in) {
  id = in.readString();
  username = in.readString();
  nameFirst = in.readString();
  nameLast = in.readString();
  avatar = in.readString();
  createdAt = (Date) in.readSerializable();
  modifiedAt = (Date) in.readSerializable();
 }
 
 public static final Parcelable.Creator<User> CREATOR = new Parcelable.Creator<User>() {
  
        public User createFromParcel(Parcel in) {
            return new User(in);
        }
 
        public User[] newArray(int size) {
            return new User[size];
        }
        
    };

}

Message.java

package com.acme.social.api;

import java.util.Date;

import android.os.Parcel;
import android.os.Parcelable;

public class Message implements Parcelable {
    
 private String id;
    private String userId;
    private String body;
    private Date createdAt;
    private Date modifiedAt;
    // database related
    protected String userUsername;
 protected String userNameFirst;
 protected String userNameLast;
 protected String userAvatar;
    
    public Message(String id, String userId, String body, Date createdAt, Date modifiedAt) {
     this.id = id;
        this.userId = userId;
        this.body = body;
        this.createdAt = createdAt;
        this.modifiedAt = modifiedAt;
    }
    
    public Message(String id, String userId, String body, Date createdAt, Date modifiedAt, String userUsername, String userNameFirst, String userNameLast, String userAvatar) {
     this.id = id;
     this.userId = userId;
        this.body = body;
        this.createdAt = createdAt;
        this.modifiedAt = modifiedAt;
  this.userUsername = userUsername;
  this.userNameFirst = userNameFirst;
  this.userNameLast = userNameLast;
  this.userAvatar = userAvatar;
 }
    
    public Message(Parcel in) {
     readFromParcel(in);
    }
    
    public String getId() {
        return id;
    }
   
    public String getBody() {
        return body;
    }

    public Date getCreatedAt() {
        return createdAt;
    }

    public Date getModifiedAt() {
        return modifiedAt;
    }

    public String getUserId() {
        return userId;
    }
    
    public String getUserUsername() {
  return userUsername;
 }
    
    public String getUserNameFirst() {
  return userNameFirst;
 }
    
    public String getUserNameLast() {
  return userNameLast;
 }
 
 public String getUserName() {
  return userNameFirst + " " + userNameLast;
 }

 public String getUserAvatar() {
  return userAvatar;
 }
 
 public void setUserId(String userId) {
  this.userId = userId;
 }

    @Override
 public int describeContents() {
  return 0;
 }

 @Override
 public void writeToParcel(Parcel out, int flags) {
  out.writeString(id);
  out.writeString(userId);
  out.writeString(userUsername);
  out.writeString(userNameFirst);
  out.writeString(userNameLast);
  out.writeString(userAvatar);
  out.writeString(body);
  out.writeSerializable(createdAt);
  out.writeSerializable(modifiedAt);
 }
 
 private void readFromParcel(Parcel in) {
  id = in.readString();
  userId = in.readString();
  userUsername = in.readString();
  userNameFirst = in.readString();
  userNameLast = in.readString();
  userAvatar = in.readString();
  body = in.readString();
  createdAt = (Date) in.readSerializable();
  modifiedAt = (Date) in.readSerializable();
 }
 
 public static final Parcelable.Creator<Message> CREATOR = new Parcelable.Creator<Message>() {
  
        public Message createFromParcel(Parcel in) {
            return new Message(in);
        }
 
        public Message[] newArray(int size) {
            return new Message[size];
        }
        
    };
    
}

Conversation.java

package com.acme.social.api;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import android.os.Parcel;
import android.os.Parcelable;

public class Conversation implements Parcelable {
    
    private String id;
    private User fromUser;
    private User toUser;
    private Date createdAt;
    private Date modifiedAt;
    private String repliedId;
    private Date repliedAt;
    private String repliedBy;
    private String repliedBody;
    private String avatar;
    private List<Message> messages;
    
    public Conversation(String id, User fromUser, User toUser, Date createdAt, Date modifiedAt, String repliedId, Date repliedAt, String repliedBy, String repliedBody) {
        this.id = id;
        this.fromUser = fromUser;
        this.toUser = toUser;
        this.createdAt = createdAt;
        this.modifiedAt = modifiedAt;
        this.repliedId = repliedId;
        this.repliedAt = repliedAt;
        this.repliedBy = repliedBy;
        this.repliedBody = repliedBody;
    }
    
    public Conversation(Parcel in) {
     readFromParcel(in);
    }

    public Date getCreatedAt() {
        return createdAt;
    }

    public User getFromUser() {
        return fromUser;
    }

    public String getId() {
        return id;
    }

    public List<Message> getMessages() {
        return messages;
    }
    
    public Message getLastMessage() {
     if (messages != null && !messages.isEmpty()) {
      return messages.get(messages.size() - 1);
  }
     return null;
    }

    public Date getModifiedAt() {
        return modifiedAt;
    }
    
    public String getRepliedId() {
        return repliedId;
    }

    public Date getRepliedAt() {
        return repliedAt;
    }

    public String getRepliedBy() {
        return repliedBy;
    }
    
    public String getRepliedBody() {
     return repliedBody;
    }

    public User getToUser() {
        return toUser;
    }
    
    
 public String getAvatar() {
  return avatar;
 }

 public void getAvatar(String avatar) {
  this.avatar = avatar;
 }

 @Override
 public int describeContents() {
  return 0;
 }

 @Override
 public void writeToParcel(Parcel out, int flags) {
  out.writeString(id);
  out.writeParcelable(fromUser, flags);
  out.writeParcelable(toUser, flags);
  out.writeSerializable(createdAt);
  out.writeSerializable(modifiedAt);
  out.writeSerializable(repliedAt);
  out.writeList(messages);
 }
 
 private void readFromParcel(Parcel in) {
     id = in.readString();
     fromUser = in.readParcelable(User.class.getClassLoader());
     toUser = in.readParcelable(User.class.getClassLoader());
     createdAt = (Date) in.readSerializable();
     modifiedAt = (Date) in.readSerializable();
     repliedAt = (Date) in.readSerializable();
     messages = new ArrayList<Message>();
     in.readList(messages, Message.class.getClassLoader());
 }
 
 public static final Parcelable.Creator<Conversation> CREATOR = new Parcelable.Creator<Conversation>() {
  
        public Conversation createFromParcel(Parcel in) {
            return new Conversation(in);
        }
 
        public Conversation[] newArray(int size) {
            return new Conversation[size];
        }
        
    };
    
    @Override
    public boolean equals(Object obj) {
        if (!(obj instanceof Conversation)) {
           return false;
        }
        return this.getId().equals(((Conversation) obj).getId());
    }

}

You may have noticed the Parcelable interface implemented by all these entities. This is required because all entity class instances will be generated and returned by an IntentService as I will explain in my next article. If you don't know what the Parcelable interface is used for, have a look at my previous A Parcelable Tutorial for Android article.

In addition to the above entity classes, we also need to create a Profile class which represents the current authenticated user:

Profile.java

package com.acme.social.api;

import java.util.Date;
import java.util.List;

import android.os.Parcel;
import android.os.Parcelable;

public class Profile implements Parcelable {
    
    protected String id;
    protected String nameFirst;
    protected String nameLast;
    protected String username;
    protected String email;
    protected String avatar;
    protected Date createdAt;
    protected Date modifiedAt;
    
    public Profile(String id, String nameFirst, String nameLast, String username, String email, String avatar, Date createdAt, Date modifiedAt) {
        this.id = id;
        this.nameFirst = nameFirst;
        this.nameLast = nameLast;
        this.username = username;
        this.email = email;
        this.avatar = avatar;
        this.createdAt = createdAt;
        this.modifiedAt = modifiedAt;
    }

    public Profile(Parcel in) {
        readFromParcel(in);
    }
    
    public String getAvatar() {
        return avatar;
    }

    public Date getCreatedAt() {
        return createdAt;
    }

    public String getId() {
        return id;
    }

    public Date getModifiedAt() {
        return modifiedAt;
    }

    public String getNameFirst() {
        return nameFirst;
    }

    public String getNameLast() {
        return nameLast;
    }

    public String getUsername() {
        return username;
    } 
    
    public String getEmail() {
        return email;
    } 

 @Override
 public int describeContents() {
  return 0;
 }

 @Override
 public void writeToParcel(Parcel out, int flags) {
  out.writeString(id);
  out.writeString(nameFirst);
  out.writeString(nameLast);
  out.writeString(username);
  out.writeString(email);
  out.writeString(avatar);
  out.writeSerializable(createdAt);
  out.writeSerializable(modifiedAt);
 }
 
 private void readFromParcel(Parcel in) {
  id = in.readString();
  nameFirst = in.readString();
  nameLast = in.readString();
  username = in.readString();
  email = in.readString();
  avatar = in.readString();
  createdAt = (Date) in.readSerializable();
  modifiedAt = (Date) in.readSerializable();
 }
 
 public static final Parcelable.Creator<Profile> CREATOR = new Parcelable.Creator<Profile>() {
  
        public Profile createFromParcel(Parcel in) {
            return new Profile(in);
        }
 
        public Profile[] newArray(int size) {
            return new Profile[size];
        }
        
    };
    
    public User getUser() {
     return new User(id, username, nameFirst, nameLast, avatar, createdAt, modifiedAt);
    }

}

JSON Mapping

Now that our entity classes are defined, we can move onto creating our mapping classes inside the com.acme.social.api.json package. It is pretty straight-forward Jackson configuration as illustrated below:

UserMixin.java

package com.acme.social.api.json;

import java.util.Date;

import org.codehaus.jackson.annotate.JsonCreator;
import org.codehaus.jackson.annotate.JsonIgnoreProperties;
import org.codehaus.jackson.annotate.JsonProperty;
import org.codehaus.jackson.map.annotate.JsonSerialize;

@JsonIgnoreProperties(ignoreUnknown = true)
public class UserMixin {
    
    @JsonCreator
 UserMixin(
  @JsonProperty("id") String id, 
  @JsonProperty("username") String username, 
        @JsonProperty("name_first") String nameFirst, 
        @JsonProperty("name_last") String nameLast, 
        @JsonProperty("avatar") String avatar,
        @JsonProperty("created_at") Date createdAt,
        @JsonProperty("modified_at") Date modifiedAt
    ) {}
    
    @JsonProperty("created_at")
 @JsonSerialize(using = DateSerializer.class)
 Date createdAt;
       
    @JsonProperty("modified_at")
 @JsonSerialize(using = DateSerializer.class)
 Date modifiedAt;
}

MessageMixin.java

package com.acme.social.api.json;

import java.util.Date;

import org.codehaus.jackson.annotate.JsonCreator;
import org.codehaus.jackson.annotate.JsonIgnoreProperties;
import org.codehaus.jackson.annotate.JsonProperty;
import org.codehaus.jackson.map.annotate.JsonSerialize;

@JsonIgnoreProperties(ignoreUnknown = true)
public class MessageMixin {
    
    @JsonCreator
 MessageMixin(
  @JsonProperty("id") String id, 
        @JsonProperty("user_id") String userId, 
        @JsonProperty("body") String body,
        @JsonProperty("created_at") Date createdAt,
        @JsonProperty("modified_at") Date modifiedAt
    ) {}
    
    @JsonProperty("created_at")
 @JsonSerialize(using = DateSerializer.class)
 Date createdAt;
       
    @JsonProperty("modified_at")
 @JsonSerialize(using = DateSerializer.class)
 Date modifiedAt;
   
}

ConversationMixin.java

package com.acme.social.api.json;

import java.io.IOException;
import java.util.Date;
import java.util.List;

import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.JsonParser;
import org.codehaus.jackson.JsonProcessingException;
import org.codehaus.jackson.annotate.JsonCreator;
import org.codehaus.jackson.annotate.JsonIgnoreProperties;
import org.codehaus.jackson.annotate.JsonProperty;
import org.codehaus.jackson.map.DeserializationContext;
import org.codehaus.jackson.map.JsonDeserializer;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.annotate.JsonDeserialize;
import org.codehaus.jackson.map.annotate.JsonSerialize;
import org.codehaus.jackson.type.TypeReference;

import com.acme.social.api.Message;
import com.acme.social.api.User;

@JsonIgnoreProperties(ignoreUnknown = true)
public class ConversationMixin {
    
    @JsonCreator
 ConversationMixin(
        @JsonProperty("id") String id, 
        @JsonProperty("from_user") User fromUser,
        @JsonProperty("to_user") User toUser,
        @JsonProperty("created_at") Date createdAt,
        @JsonProperty("modified_at") Date modifiedAt,
        @JsonProperty("replied_id") String repliedId,
        @JsonProperty("replied_at") Date repliedAt,
        @JsonProperty("replied_by") String repliedBy,
        @JsonProperty("replied_body") String repliedBody
    ) {}
    
    @JsonProperty("created_at")
 @JsonSerialize(using = DateSerializer.class)
 Date createdAt;
       
    @JsonProperty("modified_at")
 @JsonSerialize(using = DateSerializer.class)
 Date modifiedAt;
    
    @JsonProperty("replied_at")
 @JsonSerialize(using = DateSerializer.class)
 Date repliedAt;
   
    @JsonProperty("messages")
    @JsonDeserialize(using = MessageListDeserializer.class)
    List<Message> messages;
    
    private static class MessageListDeserializer extends JsonDeserializer<List<Message>> {
        
        @SuppressWarnings("unchecked")
        @Override
        public List<Message> deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
            
            ObjectMapper mapper = new ObjectMapper();
            mapper.setDeserializationConfig(ctxt.getConfig());
            jp.setCodec(mapper);
            
            if(jp.hasCurrentToken()) {
                JsonNode dataNode = jp.readValueAsTree();
                if(dataNode != null) {
                    return (List<Message>) mapper.readValue(dataNode, new TypeReference<List<Message>>() {});
                }
            }

            return null;
        }
        
    }
}

ProfileMixin.java

package com.acme.social.api.json;

import java.io.IOException;
import java.util.Date;
import java.util.List;

import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.JsonParser;
import org.codehaus.jackson.JsonProcessingException;
import org.codehaus.jackson.annotate.JsonCreator;
import org.codehaus.jackson.annotate.JsonIgnoreProperties;
import org.codehaus.jackson.annotate.JsonProperty;
import org.codehaus.jackson.map.DeserializationContext;
import org.codehaus.jackson.map.JsonDeserializer;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.annotate.JsonDeserialize;
import org.codehaus.jackson.map.annotate.JsonSerialize;
import org.codehaus.jackson.type.TypeReference;

@JsonIgnoreProperties(ignoreUnknown = true)
public class ProfileMixin {
    
    @JsonCreator
    ProfileMixin(
  @JsonProperty("id") String id, 
        @JsonProperty("name_first") String nameFirst, 
        @JsonProperty("name_last") String nameLast, 
        @JsonProperty("username") String username, 
        @JsonProperty("email") String email, 
        @JsonProperty("avatar") String avatar,
        @JsonProperty("created_at") Date createdAt,
        @JsonProperty("modified_at") Date modifiedAt
    ) {}
    
    @JsonProperty("created_at")
 @JsonSerialize(using = DateSerializer.class)
 Date createdAt;
       
    @JsonProperty("modified_at")
 @JsonSerialize(using = DateSerializer.class)
 Date modifiedAt;
}

You may have noticed the DateSerializer.class references. This custom serializer class is necessary because we need to generate an ISO8601 formatted dates when serializing our entities. This is required not only for consistency purposes - the dates we receive are in this format - but also the format needs to be recognizable by PHP's DateTime constructor when serialized JSON is unmarshalled on the server.

DateSerializer.java

package com.acme.social.api.json;

import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;

import org.codehaus.jackson.JsonGenerator;
import org.codehaus.jackson.JsonProcessingException;
import org.codehaus.jackson.map.JsonSerializer;
import org.codehaus.jackson.map.SerializerProvider;

public class DateSerializer extends JsonSerializer<Date> {
 @Override
 public void serialize(Date value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
  SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ"); // iso8601
  String formattedDate = formatter.format(value);
  jgen.writeString(formattedDate);
 }
}

Finally, let's couple our entity classes with their corresponding mixin classes. We will utilize a module class to accomplish this as illustrated below:

AcmeModule.java

package com.acme.social.api.json;

import org.codehaus.jackson.Version;
import org.codehaus.jackson.map.module.SimpleModule;

import com.acme.social.api.Conversation;
import com.acme.social.api.Message;
import com.acme.social.api.Profile;
import com.acme.social.api.User;

public class AcmeModule extends SimpleModule {
    
    public AcmeModule() {
  super("AcmeModule", new Version(1, 0, 0, null));
 }
    
    @Override
 public void setupModule(SetupContext context) {    
  context.setMixInAnnotations(User.class, UserMixin.class);
  context.setMixInAnnotations(Profile.class, ProfileMixin.class);
  context.setMixInAnnotations(Conversation.class, ConversationMixin.class);
  context.setMixInAnnotations(Message.class, MessageMixin.class);
    }
}

Template Classes

Template classes contain all our API methods to interact with the remote server. We will create a template class for each of our entities (unless they are embedded).

UserTemplate.java

package com.acme.social.api;

public class UserTemplate extends BaseTemplate {
    
 private final AcmeTemplate api;
    
    public UserTemplate(AcmeTemplate api, boolean isAuthorized) {
        super(isAuthorized);
        this.api = api;
    }

}

ConversationTemplate.java

package com.acme.social.api;

import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.List;

public class ConversationTemplate extends BaseTemplate {
    
 private final AcmeTemplate api;
    
    public ConversationTemplate(AcmeTemplate api, boolean isAuthorized) {
        super(isAuthorized);
        this.api = api;
    }
    
    public List<Conversation> getConversations(String userId) {
     return api.fetchCollection("/users/" + userId + "/conversations.json?limit=" + ConversationTemplate.DEFAULT_LIMIT, Conversation.class, null);
 }
    
    public Conversation getConversation(String userId, String conversationId) {
     return api.fetchObject("/users/" + userId + "/conversations/" + conversationId + ".json", Conversation.class, null);
    }
    
    public List<Message> getMessages(String userId, String conversationId) {
     return api.fetchCollection("/users/" + userId + "/conversations/" + conversationId + "/messages.json", Message.class, null);
    }
    
    public Message postMessage(String userId, String conversationId, Message message) {
     return api.postForObject("/users/" + userId + "/conversations/" + conversationId + "/messages.json", message, Message.class);
    }
    
}

ProfileTemplate.java

package com.acme.social.api;

public class ProfileTemplate extends BaseTemplate {
    
 private final AcmeTemplate api;
    
    public ProfileTemplate(AcmeTemplate api, boolean isAuthorized) {
        super(isAuthorized);
        this.api = api;
    }
    
    public Profile getProfile() {
     return api.fetchObject("/profiles/me.json", Profile.class, null);
    }

    
}

Finally, we create a base template class called AcmeTemplate to tie everything together. This class contains a couple of wrapper methods (i.e. postForObject, fetchObject, fetchCollection) - not a complete list but enough to serve the purpose of this article - to handle URL generation. It also contains a method called registerAcmeJsonModule which binds our JSON setup (AcmeModule) to the REST template provided by Spring for Android and ensures that the property naming strategy is correct (lowercase words separated with underscores).

AcmeTemplate.java

package com.acme.social.api;

import java.io.IOException;
import java.util.List;

import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.PropertyNamingStrategy;
import org.codehaus.jackson.map.type.CollectionType;
import org.codehaus.jackson.map.type.TypeFactory;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.http.converter.json.MappingJacksonHttpMessageConverter;
import org.springframework.social.UncategorizedApiException;
import org.springframework.social.oauth2.AbstractOAuth2ApiBinding;
import org.springframework.social.support.ClientHttpRequestFactorySelector;
import org.springframework.social.support.URIBuilder;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.DefaultResponseErrorHandler;
import org.springframework.web.client.RestTemplate;

import com.acme.social.api.json.AcmeModule;

public class AcmeTemplate extends AbstractOAuth2ApiBinding implements Acme {
    
    private static final String ACME_API_URL = "http://10.0.2.2/app_dev.php/api";
 private UserTemplate userOperations;
 private ProfileTemplate profileOperations;
    private ConversationTemplate conversationOperations;
    
    private ObjectMapper objectMapper;
    
    public AcmeTemplate() {
        initialize();  
    }
    
    public AcmeTemplate(String accessToken) {
  super(accessToken);
  initialize();
 }
    
    public void initSubApis() {
        this.userOperations = new UserTemplate(this, isAuthorized());
        this.profileOperations = new ProfileTemplate(this, isAuthorized());
        this.conversationOperations = new ConversationTemplate(this, isAuthorized());
    }
    
    public UserTemplate userOperations() {
        return this.userOperations;
    }
    
    public ProfileTemplate profileOperations() {
        return this.profileOperations;
    }
    
    public ConversationTemplate conversationOperations() {
        return this.conversationOperations;
    }
    
    private void initialize() {
  registerAcmeJsonModule(getRestTemplate());
  getRestTemplate().setErrorHandler(new DefaultResponseErrorHandler());
  // Wrap the request factory with a BufferingClientHttpRequestFactory so that the error handler can do repeat reads on the response.getBody()
  super.setRequestFactory(ClientHttpRequestFactorySelector.bufferRequests(getRestTemplate().getRequestFactory()));
  initSubApis();
 }
  
 private void registerAcmeJsonModule(RestTemplate restTemplate2) {
        
  this.objectMapper = new ObjectMapper();    
  this.objectMapper.registerModule(new AcmeModule());
  this.objectMapper.setPropertyNamingStrategy(new PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy());
        
  List<HttpMessageConverter<?>> converters = getRestTemplate().getMessageConverters();
  for (HttpMessageConverter<?> converter : converters) {
   if(converter instanceof MappingJacksonHttpMessageConverter) {
    MappingJacksonHttpMessageConverter jsonConverter = (MappingJacksonHttpMessageConverter) converter;
    jsonConverter.setObjectMapper(objectMapper);
   }
  }
 }
 
 public <T> T postForObject(String path, Object request, Class<T> responseType) {
  URIBuilder uriBuilder = URIBuilder.fromUri(ACME_API_URL + path);
  return getRestTemplate().postForObject(uriBuilder.build(), request, responseType);
 }
 
 public <T> T fetchObject(String path, Class<T> type, MultiValueMap<String, String> queryParameters) {
  URIBuilder uriBuilder = URIBuilder.fromUri(ACME_API_URL + path);
  if (queryParameters != null) {
   uriBuilder.queryParams(queryParameters);
  }
  return getRestTemplate().getForObject(uriBuilder.build(), type);
 }
 
 public <T> List<T> fetchCollection(String path, Class<T> type, MultiValueMap<String, String> queryParameters) {
  URIBuilder uriBuilder = URIBuilder.fromUri(ACME_API_URL + path);
  if (queryParameters != null) {
   uriBuilder.queryParams(queryParameters);
  }
  JsonNode dataNode = getRestTemplate().getForObject(uriBuilder.build(), JsonNode.class);
  return deserializeDataList(dataNode, type);
 }
 
 @SuppressWarnings("unchecked")
 private <T> List<T> deserializeDataList(JsonNode jsonNode, final Class<T> elementType) {
  try {
   CollectionType listType = TypeFactory.defaultInstance().constructCollectionType(List.class, elementType);
   return (List<T>) objectMapper.readValue(jsonNode, listType);
  } catch (IOException e) {
   throw new UncategorizedApiException("Error deserializing data from: " + e.getMessage(), e);
  }
 }

}

Connection Classes

Before we go into code deatils, have a look at the below UML class diagram that explains our setup. We are basically going to model our design after existing packages already provided by Spring Social; see the Facebook package by Spring Social for reference.

Spring Social - UML Diagram

In the above UML class diagram, light green boxes represent our custom classes while the yellow ones represent classes provided by Spring Social.

AcmeConnectionFactory
This class extends the OAuth2ConnectionFactory class. As you can already infer from the naming convention, this class is responsible for generating an OAuth2Connection. Its constructor injects our own custom AcmeServiceProvider and AcmeAdapter class instances into the base connection factory class.

AcmeConnectionFactory.java

package com.acme.social.connect;

import com.acme.social.api.Acme;
import org.springframework.social.connect.support.OAuth2ConnectionFactory;

public class AcmeConnectionFactory extends OAuth2ConnectionFactory<Acme> {

 public AcmeConnectionFactory(String clientId, String clientSecret, String authorizeUrl, String tokenUrl) {
  super("Acme", new AcmeServiceProvider(clientId, clientSecret, authorizeUrl, tokenUrl), new AcmeAdapter());
 }

}

AcmeAdapter
As can be seen in the UML diagram above, AcmeAdapter is a component of OAuth2ConnectionFactory and OAuth2Connection classes. This allows us to inject our authenticated user information (i.e. id, username, and avatar) to the active connection so that we can extract authenticated user details from the connection later when needed in the application. In fact, if you examine the SQLite database table containing the connection information after a successful authentication, you can see that there is a column for provider user id and this class serves as the key mechanism to populate that column.

AcmeAdapter.java

package com.acme.social.connect;

import com.acme.social.api.Acme;
import com.acme.social.api.Profile;

import org.springframework.social.ApiException;
import org.springframework.social.connect.ApiAdapter;
import org.springframework.social.connect.ConnectionValues;
import org.springframework.social.connect.UserProfileBuilder;

public class AcmeAdapter implements ApiAdapter<Acme> {
    
    @Override
    public boolean test(Acme acme) {
  try {
   return true;
  } catch (ApiException e) {
            e = null;
   return false;
  }
 }

    @Override
 public void setConnectionValues(Acme acme, ConnectionValues values) {
  Profile profile = acme.profileOperations().getProfile();
  values.setProviderUserId(profile.getId());
  values.setDisplayName(profile.getUsername());
  values.setImageUrl(profile.getAvatar());
 }

    @Override
 public org.springframework.social.connect.UserProfile fetchUserProfile(Acme acme) {
     
  Profile profile = acme.profileOperations().getProfile();
        
  return new UserProfileBuilder().setName(profile.getUsername())
            .setFirstName(profile.getNameFirst())
            .setLastName(profile.getNameLast())
            .setEmail(profile.getEmail())
            .setUsername(profile.getUsername())
            .build();
        
 }
 
    @Override
 public void updateStatus(Acme acme, String message) {
  acme.profileOperations().updateStatus(message);
 }

}

AcmeServiceProvider
This class extends AbstractOAuth2ServiceProvider and is responsible for binding our AcmeTemplate class with the OAuth2Template class that makes OAuth2 calls using the provided REST template.

AcmeServiceProvider.java

package com.acme.social.connect;

import com.acme.social.api.acme;
import com.acme.social.api.AcmeTemplate;
import org.springframework.social.oauth2.AbstractOAuth2ServiceProvider;
import org.springframework.social.oauth2.OAuth2Template;

public class AcmeServiceProvider extends AbstractOAuth2ServiceProvider<Acme> {

 public AcmeServiceProvider(String clientId, String clientSecret, String authorizeUrl, String tokenUrl) {
  super(new OAuth2Template(clientId, clientSecret, authorizeUrl, tokenUrl));
 }
    
    @Override
 public Acme getApi(String accessToken) {
  return new AcmeTemplate(accessToken);
 }
 
}

And with this last class defined, our setup is complete. In my next article, I will explain how to use these classes inside an Android application.

Tuesday, September 11, 2012

A Parcelable Tutorial for Android

Parcelable Interface Overview

In one of my earlier posts, I mentioned writing an article about FOSOAuthBundle integration with an Android client. To keep that article to the point, I need to explain some concepts beforehand. One of the important concepts is the Android Parcelable interface that allows data to be transferred between different processes/threads. Certain network operations with Android such as authentication with OAuth2 and then fetching data from a REST endpoint should be performed in the background in order not to block the UI thread. This requires data to be fetched by a service (I have opted for Intent Services in my implementation) in the background and then passed back to the calling activity/fragment with a result callback. This is where the Parcelable interface comes into play.

Basically, the Parcelable interface allows your classes to be flattened inside a message container called a Parcel to facilitate high performance inter process communication. The received parcel data can then be unflattened to generate object/entity instances.

A Basic Parcelable Example

A Parcelable implementation is pretty straight forward. Override the necessary methods called writeToParcel() and describeContents(), add a static field called CREATOR which generates instances of your Parcelable class from a Parcel, and overload the class constructor which expects a Parcel as a parameter and calls the readFromParcel() utility method. Here is a basic example:

public class Conversation implements Parcelable {

    // ...
    protected String lastComment;
    protected Integer messageCount;
    protected Date createdAt;
    // ...

    // ...
    public Conversation(Parcel in) {
     readFromParcel(in);
    }

    @Override
    public void writeToParcel(Parcel out, int flags) {
        // ...
        out.writeString(lastComment);
        out.writeInt(messageCount);
 out.writeSerializable(createdAt);
        // ...
    }

    private void readFromParcel(Parcel in) {  
        // ...
        lastComment = in.readString();
 messageCount = in.readInt();
 createdAt = (Date) in.readSerializable();
        // ...
    }

    public static final Parcelable.Creator CREATOR = new Parcelable.Creator() {
  
        public Conversation createFromParcel(Parcel in) {
            return new Conversation(in);
        }
 
        public Conversation[] newArray(int size) {
            return new Conversation[size];
        }
        
    };

    @Override
    public int describeContents() {
        return 0;
    }

}

Nested Parcelable Classes

Lets say we have a Conversation class instance with an embedded User class instance that holds some information about a user that has initiated the conversation. Basically, we are talking about an one-to-one embedded mapping. In this case, if the fromUser property holds a User instance, then writing the user data to parcel would be accomplished as shown below:

@Override
public void writeToParcel(Parcel out, int flags) {
    // ...
    out.writeParcelable(fromUser, flags);
    // ...
}

Reading the user data from parcel:

private void readFromParcel(Parcel in) {  
    // ...
    fromUser = in.readParcelable(User.class.getClassLoader());
    // ...
}

If you have a one-to-many embedded relationship such as a list of messages in a conversation, then the syntax would change as follows:

@Override
public void writeToParcel(Parcel out, int flags) {
    // ...
    out.writeList(messages);
    // ...
}

Reading from the parcel:

private void readFromParcel(Parcel in) {  
    // ...
    in.readList(messages, Message.class.getClassLoader());
    // ...
}

Boolean Types

Android API does not have a method to write a single boolean value to a parcel. In this case, you can utilize the writeInt() method as shown below:

@Override
public void writeToParcel(Parcel out, int flags) {
    // ...
    out.writeInt(booleanValue ? 1 : 0);
    // ...
}

Reading from the parcel:

private void readFromParcel(Parcel in) {  
    // ...
    booleanValue = in.readInt() == 1;
    // ...
}

Enum Types

To flatten an Enum type in a parcel, simply implement the Parcelable interface for the Enum type. Here is an example:

public enum Status implements Parcelable {
  
    STARTED, PAUSED, FINISHED;
  
    public static final Parcelable.Creator CREATOR = new Parcelable.Creator() {
   
        public Status createFromParcel(Parcel in) {
            return Status.values()[in.readInt()];
 }
 
        public Status[] newArray(int size) {
     return new Status[size];
 }
         
    };

    @Override
    public int describeContents() {
 return 0;
    }

    @Override
    public void writeToParcel(Parcel out, int flags) {
 out.writeInt(ordinal());
    }

}

When writing to the parcel, treat it as a nested Parcelable class:

@Override
public void writeToParcel(Parcel out, int flags) {
    // ...
    out.writeParcelable(status, flags);
    // ...
}

Reading from the parcel:

private void readFromParcel(Parcel in) {  
    // ...
    status = in.readParcelable(Status.class.getClassLoader());
    // ...
}

Wednesday, August 15, 2012

Atomicly update an embedded document with Doctrine MongoDB ODM

I had to deal with an unexpected problem recently. In my setup, I have a Conversation document which contains multiple Message documents embedded inside - a one-to-many mapping basically - and I need to atomically push a new Message when a user replies to a conversation. Below is the initial code that I implemented:

public function reply($conversationId, Message $message, $flush = true)
{            
    $this->dm->createQueryBuilder($this->class)
        ->update()
        ->field('archivers')->unsetField()
        ->field('repliedBy')->set($message->getUserId())
        ->field('repliedBody')->set($message->getBody())
        ->field('repliedAt')->set(new \DateTime())
        ->field('modifiedAt')->set(new \DateTime())
        ->field('messages')->push($message)
        ->field('id')->equals(new \MongoId($conversationId))
        ->getQuery()
        ->execute();
}

Unfortunately, this approach failed miserably. Because if you use the push() call with a document instance, you get a Catchable Fatal Error: Object of class ... could not be converted to string in /vendor/bundles/Symfony/Bundle/DoctrineMongoDBBundle/Logger/DoctrineMongoDBLogger.php line 280 error.

After losing some hair over this, I decided to seek help. Luckily, jmikola (one of the developers of Doctrine MongoDB ODM) replied to my question at stackoverflow.com.

Turns out, the query builder is not really intended to work with document instances. As an alternative approach, I used an array representation of the Message document as illustrated below:

public function reply($conversationId, Message $message)
{            
    $this->dm->createQueryBuilder($this->class)
        ->update()
        ->field('archivers')->unsetField()
        ->field('repliedBy')->set($message->getUserId())
        ->field('repliedBody')->set($message->getBody())
        ->field('repliedAt')->set(new \DateTime())
        ->field('modifiedAt')->set(new \DateTime())
        ->field('messages')->push(array(
            '_id' => new \MongoId(),
            'userId' => $message->getuserId(),
            'body' => $message->getBody(),
            'createdAt' => new \DateTime(),
            'modifiedAt' => new \DateTime(),
        ))
        ->field('id')->equals(new \MongoId($conversationId))
        ->getQuery() 
        ->execute();
}

According to jmikola, you can also use the document manager to achieve the same result (see his last comment under his answer), but I have not verified this yet.

Saturday, August 4, 2012

Updates to managed documents under Doctrine MongoDB ODM

If you update a document returned by Doctrine MongoDB ODM, changes to that document will be flushed to even if you do not call the persist() method on that document. That is because the document returned is a managed document. In fact, the persist() call gets ignored for these managed documents. From the documentation:

If X is a preexisting managed document, it is ignored by the persist operation. However, the persist operation is cascaded to documents referenced by X, if the relationships from X to these other documents are mapped with cascade=PERSIST or cascade=ALL.

In my case, this recently caused an issue where duplicate embedded documents ended up being pushed into a parent document because I was using and updating the parent document for generating a custom Symfony2 form. Turned out to be a bad idea... If you need to update these type of documents, detach them first.

Sunday, July 1, 2012

Recent News and Articles - July 2012

Google announced the Google Compute Engine. Even though it is too early to comment, as a customer of Amazon's EC2 services, I am loving this; competition is definitely good news for business.

For the gamers out there, Epic Games introduced the Unreal Engine 4 recently. Have a look at the Wired article to understand the implications of this new engine on the gaming and hardware industry. Here is the development walk-through. There is also a showcase demo available for a preview.

Warning, shameless self promotion here: the Symfony Blog picked my article about creating a custom JMSSerializerBundle handler.

Thursday, June 21, 2012

Creating a Custom JMSSerializerBundle Handler

A couple of days ago, I had to create a custom JMSSerializerBundle handler to process the serialization of a certain property of one of my models. In my case, I have a user model that has a single associated avatar stored on Amazon S3. The problem is that this avatar URI consists of just the path section (for example: /avatars/12/34/43/123443.jpg) instead of the full URL. This is required to allow my application to work with multiple buckets, each of which corresponds to a different environment such as development, production, etc.

However, the Android client I am currently working on (or any other REST client for that matter) needs the full URL of each avatar returned from the server. Injecting this logic into the model class was obviously not a really good idea, so I decided to write a custom JMSSerializerBundle handler to achieve my objective. The idea is the have the custom handler access the required system services and transform the avatar path from a relative on to a full one. here is how I have implemented it.

First, create your custom handler class. At this point, we do not care about de-serialization so our class will only contain serialization logic.

src/Acme/AmazonBundle/Serializer/AvatarHandler.php

use Symfony\Component\Yaml\Inline;
use JMS\SerializerBundle\Serializer\YamlSerializationVisitor;
use JMS\SerializerBundle\Serializer\GenericSerializationVisitor;
use JMS\SerializerBundle\Serializer\JsonSerializationVisitor;
use JMS\SerializerBundle\Serializer\XmlSerializationVisitor;
use JMS\SerializerBundle\Serializer\VisitorInterface;
use JMS\SerializerBundle\Serializer\Handler\SerializationHandlerInterface;
use Acme\AmazonBundle\Core\Entity\S3\AvatarManager;
use Acme\AmazonBundle\Core\Entity\S3\Avatar;

class AvatarHandler implements SerializationHandlerInterface
{
    protected $avatarManager;

    public function __construct(AvatarManager $avatarManager)
    {
        $this->avatarManager = $avatarManager;
    }

    public function serialize(VisitorInterface $visitor, $data, $type, &$visited)
    {
        if (!$data instanceof Avatar) {
            return;
        }
                
        $data = sprintf('http://%s/%s', $this->avatarManager->getDomain(), $this->avatarManager->getTarget($data));
                
        if ($visitor instanceof XmlSerializationVisitor) {
            
            if (null === $visitor->document) {
                $visitor->document = $visitor->createDocument(null, null, true);
            }
            
            $visited = true;
            return $visitor->document->createTextNode($data);
            
        } else if ($visitor instanceof GenericSerializationVisitor) {
            
            $visited = true;
            return $data;
            
        } else if ($visitor instanceof YamlSerializationVisitor) {
            
            $visited = true;
            return Inline::dump($data);
            
        }
        
    }

}

Note the injected S3 Manager (AvatarManager) service which handles an S3 Entity (Avatar).

Second, create your handler factory class:

src/Acme/AmazonBundle/DependencyInjection/Factory/AvatarHandlerFactory.php

namespace Acme\AmazonBundle\DependencyInjection\Factory;

use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\Config\Definition\Builder\ArrayNodeDefinition;
use JMS\SerializerBundle\DependencyInjection\HandlerFactoryInterface;

class AvatarHandlerFactory implements HandlerFactoryInterface
{
    public function getConfigKey()
    {
        return 'avatar';
    }

    public function getType(array $config)
    {
        return self::TYPE_SERIALIZATION;
    }

    public function addConfiguration(ArrayNodeDefinition $builder)
    {
        $builder->addDefaultsIfNotSet();
    }

    public function getHandlerId(ContainerBuilder $container, array $config)
    {
        return 'acme_amazon.serializer.avatar_handler';
    }
}

Next, register your service:

src/Acme/AmazonBundle/DependencyInjection/Factory/AvatarHandlerFactory.php

<service id="acme_amazon.serializer.avatar_handler" class="Acme\AmazonBundle\Serializer\AvatarHandler">
            <argument type="service" id="acme_amazon.entity.s3.avatar_manager" />
        </service>

Update your bundle class to register your customer handler factory:

src/Acme/AmazonBundle/AcmeAmazonBundle.php

namespace Acme\AmazonBundle;

use Symfony\Component\HttpKernel\Bundle\Bundle;
use JMS\SerializerBundle\DependencyInjection\JMSSerializerExtension;
use Acme\AmazonBundle\DependencyInjection\Factory\AvatarHandlerFactory;

class AcmeAmazonBundle extends Bundle
{
    public function configureSerializerExtension(JMSSerializerExtension $ext)
    {
        $ext->addHandlerFactory(new AvatarHandlerFactory());
    }
}

Now, create a separate property in your model class to hold a reference to your S3 Entity (Avatar). Basically, in addition to an avatarPath property - which holds the path as a string - create a new avatar property that holds an instance of an S3 entity class that acts as a wrapper when the time comes to serialize this model. This is important as you can not write custom handlers for simple data types; you need an object. Next, create your getter and setter methods to deal with this new property. The getter method is particularly important as it will be used to instantiate an Avatar object from the path string.

public function getAvatar()
{
    if (!$this->avatar) {
       $this->avatar = new Avatar($this->avatarPath);
    }
        
    return $this->avatar;
}
    
public function setAvatar(Avatar $avatar)
{
    $this->avatar = $avatar;
}

Finally, ensure that your serialization configuration is complete:

src/Acme/UserBundle/Resources/config/serializer/Document.User.xml

...
<property name="avatar" type="Avatar" expose="true" access-type="public_method" read-only="true"></property>
...

The type attribute is only required for de-serialization; however, I included it in the configuration file for completeness. At this point, your serialized model should now contain a full URI.

Sunday, June 3, 2012

Android Development

I have recently started developing a new application for the Android platform. It had been a while since I experimented with Android and this really manifested itself when I started this new project a couple of months ago. Having to deal with new Android features and corresponding backwards compatibility issues, I experienced a steep learning curve. This article highlights the lessons learned during this process.

JDK Selection

I have been a Linux user since 2001 and never looked back since then. So if you are developing on Linux, make sure to replace the default OpenJDK with Oracle's. This will save you a lot of headache dealing with peculiar errors and exceptions during the initial set-up process.

Selecting an IDE

I really like NetBeans and still develop with this IDE when it comes to PHP; it definitely feels much more intuitive than Eclipse. However, when it comes to Android development, it did not take long for me to realize I was fighting an uphill battle. With its excellent Android Development Tools (ADT) plugin integration, Eclipse definitely shines when it comes to programming for Android. In my case, the process of switching from NetBeans to Eclipse definitely took its toll both in terms of learning curve and time spent for customization to get Eclipse to a usable state.

Android API Levels

What API level to support? Well, the answer is not hard to find. On the official Android site, you can find the current statistics for platform versions. To penetrate the majority of the Android market, you need to support API Level 7; however, your application should also perform on the newest handsets running the latest version of the Android SDK at API Level 15. This is accomplished by adding the following line to your AndroidManifest.xml:

<uses-sdk android:minSdkVersion="7" android:targetSdkVersion="15" />

This simply means that when writing your code, you need to be careful in order to maintain backwards compatibility with API Level 7. This is pretty easy to accomplish thanks to Android Lint but what about forward compatibility? How do you utilize the newest features?

Supporting Libraries

New functionality introduced by the latest Android SDK will obviously not be available in older versions. This means that you will have to get comfortable utilizing the support libraries. One of these libraries is the Android Support Package that allows you to utilize certain new features such as fragments.

However, not all new functionality is supported by the Android Support Package. To get action bar support, you will need to download the ActionBarSherlock library.

These extra libraries do make the learning curve a little bit steeper but they are essential tools needed to create a Android application these days. Unfortunately, they are not silver bullets. There are still issues that need to be addressed; one of which manifests itself when you need to embed a MapView inside a fragment.

Other Libraries

If you are planning to integrate your Android application with REST services over OAuth2 , have a look at Spring for Android. In addition, you may be interested in SLF4J Android for your logging needs.

Thoughts on Android

Android reminds me of the early Linux days. The platform still has its fair share of problems but it is an open platform that is improving and maturing constantly with every release. Considering I flashed the one and only Apple product - a 5th generation IPod that was given to me as a birthday present years ago - with Rockbox the next day I received it, it is not surprising to me why I am interested in this platform. Besides, now that Android is slowly being merged into the Linux kernel, do I have a choice?