Learning Plan, Motivation and Time

In the last few months, I have been in multiple conversations around finding the time to learn. (In case you have not noticed, creating and following a learning plan is part of your personal objectives). A common suggestion I hear is to set aside half a day or a day for learning.

Earlier this week, I heard a podcast by Swizec Teller on Becoming a Senior Engineer. At the beginning of the podcast, he touched on learning. What he said consolidated ideas in my head and motivated me to write this blog.

Continue reading “Learning Plan, Motivation and Time”

Combining multiple git repositories into subdirectories of a single repository

In a few projects I worked with, the legacy code base were split into multple git repositories when they were interdependent. What I mean is that to compile the microservice deployable, you need to create the libraries contained in these multiple git repositories in a specific sequence. In addition, these libraries are not used anywhere except in the microservice. The worst culprit I encountered was a microservice that was split into six different git repositories – five Java libraries and one Java WAR file. It makes a developer’s life much easier if these six repositories are structured as a single Java maven multimodule project in a single git repository.

Obviously, you can just copy all the code to one directory and commit that. However, git allows you to copy the content from one repository into another, preserving the commit history at the same time. No one wants to lose all those change histories, right?

As an example, the code base for project x is split into three repositories called project-x-client, project-x-core and project-x-ws. You want to combine them all in project-x-ws, with each individual repository in a subdirectory. (This will later be converted into a maven submodule). After the migration, you want the code in project-x-client to be in a subdirectory called client under project-x-ws, project-x-core core, and project-x-ws in webservice.

First, you need to move all the files in project-x-core into a subdirectory called core. By doing this, when you copy all the files from this repository, the files will all be neatly located inside the directory core, instead of being under the root directory. We will use the master branch for the move.

cd /home/me/git-stuff/project-x-core
git checkout master
mkdir core
git mv src core
git mv README.md core
...
git commit -am "preparing project-x-core for migration"

You don’t need to push the change to remote. The copy can be done entirely using the local repository. To copy the project-x-core repository into project-x-ws:

cd /home/me/git-stuff/project-x-ws
git remote add r /home/me/git-stuff/project-x-core
git fetch r
git merge r/master --allow-unrelated-histories
git remote rm r

You have now pulled all the files and their commit histories into the existing repository for project-x-ws. Repeat this for client and ws. When you finish. push project-x-ws to remote to share this change with your team.

Logging outbound HTTP requests from JAX-RS client

In order to track down a bug, I needed to log HTTP requests sent from one of our web services to another third party web service. (We hosted the service, but the software was not developed in house).

Our web service was written in resteasy, a framework I was not especially familiar with. (I prefer to use the Spring stack, and always create new web services using Spring Boot). The code to call the third party web service looked like this

import javax.ws.rs.client.Invocation.Builder;
builder.buildPost(Entity.form(form)).invoke();

Surprisingly, there wasn’t an obvious way to get the request body sent. From various stackoverflow Q&A, the way to log JAX-RS outbound client requests was to create an implementation of ClientRequestFilter, and register it as a Provider in the container.

@Provider
public class MyClientRequestLoggingFilter implements ClientRequestFilter {
  private static final Logger LOG = LoggerFactory.getLogger(MyClientRequestLoggingFilter.class);	
    @Override
    public void filter(ClientRequestContext requestContext) throws IOException {
      LOG.info(requestContext.getEntity().toString());
    }
}

You then configure your web.xml to scan for providers

<context-param>
  <param-name>resteasy.scan.providers</param-name>
  <param-value>true</param-value>
</context-param>

There are quite a few warnings that because the function ClientRequestContext.getEntity() returns an Object, the default toString() may not work as expected. Unmarshalling of the object is required to log the request body.

After banging my head against a wall for an afternoon, I decided to take a completely different approach to the problem. I googled on how to enable request logging in apache httpd instead. This turned out to be a much more straightforward way to achieve what I needed. The module mod_dumpio can used to dump all input and output requests to the server into a log file. You need mod_dumpio present in the apache httpd installation. (In windows, check to see if mod_dumpio.so is in c:\apache-install-dir\modules). Stop the service, edit the httpd.conf file to include the following lines

LoadModule dumpio_module modules/mod_dumpio.so

ErrorLog "logs/error.log"
LogLevel debug
DumpIOInput On
DumpIOOutput On
LogLevel dumpio:trace7

The ErrorLog and LogLevel lines are already present in my httpd.conf. I changed the LogLevel to debug, and added the follwoing three lines to turn on the dumpio module. After server restart, all HTTP requests and responses were successfully logged to the file logs/error.log.

Lesson learnt here – if an approach turned out to be more complicated than expected, it’s worth taking a step back and rethink.

Stop accidental git commits of local dev changes to config files

During development, I often make changes to a few configuration files for local testing. Most of the time, I add each file individually into the staging area so these local config changes aren’t committed. Yesterday, I made a mistake and committed the local config. I wasn’t sure how it happened but I must have clicked the commit all tracked files accidentally. The test server was then built with my local config. Oops.

To stop this from happening again, I did some googling and found this handy git command:

git update-index --assume-unchanged <file>

This will temporarily ignore changes in the specified file. All without changing .gitignore which is a tracked file in the project.

Hot deploy JSPs in Wildfly 8.2.0

Wildfly has a development mode for JSP. In development mode, the wildfly server will check for changes in JSP files in deployed applications. JSPs can therefore be edited and tested without recompiling and redeploying the entire war file. Very handy.

This is configured in ${WILDFLY_ROOT}/standalone/configuration/standalone.xml. To enable it, set the development attribute to true in the element <jsp-config>.

<subsystem xmlns="urn:jboss:domain:undertow:1.2">
      <servlet-container name="default">
                <jsp-config development="true" tag-pooling="false"/>
            </servlet-container>
</subsystem>

If the original war file is deployed with maven, the exploded application can be find in ${WILDFLY_ROOT}/standalone/tmp/vfs/temp/tempxxxxxxx/content-xxxxxxx, where xxxxxxx is a series of random numbers. The JSPs should be in the WEB-INF directory under the application root. Replacing a JSP with a newer version and changes are immediately reflected when the webpage is reloaded.

Installing Anypoint Studio Plug-in 5.3.0 in Eclipse Mars

Installing anypoint studio should be a straight forward process, according to the official documentation on the mulesoft website. You add mulesoft to the list of available software site, select anypoint studio, click finish. Eclipse install software wizard should, theoretically, pull in required dependencies.

However, it didn’t work. I got the following exception

No repository found containing: osgi.bundle,org.eclipse.m2e.archetype.common,1.6.2.20150902-0001
….

I had a Java EE installation of Eclipse Mars (4.5). It already included maven support via m2e-wtp. Anypoint studio wasn’t happy with this version of m2e, and was also unable to pull in the required version of m2e itself. I had to manually add m2e’s update site http://download.eclipse.org/technology/m2e/releases and install m2e. After that, anypoint studio was installed without a problem.

Testing CSS selectors in Firefox

CSS selectors are used frequently when writing jquery code. Recently I discovered Firefox provides a handy syntax for testing CSS selectors:

$$('CSS selector')

In the screenshot below, I’m looking at an <h3> element with the class ‘chapter-title’ using Firefox’s web developer tools.

firefox inspector

To test if the selector ‘h3.chapter-title’ returns the expected elements, enter $$('h3.chapter-title') in the console. The command returns a nodelist with two elements, one for each match. Hovering over the individual element in the console will highlight the corresponding element in the page.

testing css selector with $$

Caching Password for Git HTTP/HTTPS connections

I got sick of entering my username and password every time I do a git operation. Luckily, git provides handy options for caching passwords. The safer option is probably to just cache the credentials in memory

git config --global credential.helper cache

This would keep the password in memory for 15 minutes. To permanently save the credentials on disk (in plain text format), use

git config --global credential.helper store

PS. I chose the later unsecure lazy option.

The Strange Default Behaviour of Git Push

It felt strange that only after a year of using git, I encountered this strange pushing logic from git. (I’m blaming it on gerrit, where pushes are always done to the review staging area using refs/for/master, instead of directly to origin/master).

My work has recently moved from svn to git. I worked on my features by creating a branch locally that tracked changes in remote master

git checkout -b feature origin/master

However when I tried to push using

git push origin/master

I got a warning along the lines of push.default is unset. Git helpfully suggested me to look at ‘git help config’. From the built in help pages and googling, I found that a simple push mode was introduced in 1.7.11. This is the default behaviour and will only push if the upstream branch’s name is the same as the local one. Because I always create a local branch using a feature name, git can’t push it to the remote server using the default behaviour.

To allow a different local branch name, I need to set the push.default config variable to upstream, which simply pushes the current branch to its upstream branch.

git config --global push.default upstream

Linked Servers : Creating a local development database in MS SQL Part 3

One of the tables I wanted to copy to my local SQL server has nearly 200 million entries. It would take far too long to copy if I generate scrips for the data using the SQL server scripting tool. Besides, I didn’t need all 200 million entries for development anyway.

The Linked Servers feature in SQL server management studio makes it simple to copy a selection of data from table to another. Once two database instances are linked, you can use SQL select and insert commands to copy data.

Create a linked server

Open the local database in SQL server management studio. Select Server Objects -> Linked Servers. Then right click and select New Linked Server. The following dialogue will appear on screen.

null

Add an appropriate name in the Linked Server text box (for example, lotsofdata-server). Under server type, select the SQL server radio button. Choose Security on the left navigation pane. Select the radio button Be made using this security context. Enter the correct username and password for the remote server.

To copy the first 10000 rows from the remote server lotsofdata-server into a table that doesn’t exist in the local database

select TOP 10000 * into dbo.[awesomeTable]  from [lotsofdata-server].[awesomeDatabase].dbo.[awesomeTable]

If the table already exist in the local database,

insert into dbo.[awesomeTable] select TOP 10000 * from [lotsofdata-server].[awesomeDatabase].dbo.[awesomeTable]
Copying an identity column

If the table you wanted to copy contains an identity column, then you need to turn on identity insert

SET IDENTITY_INSERT awesomeTable ON

and explicitly specify all the columns you are inserting into the table, like

insert into dbo.[awesomeTable] (col1, col2) select TOP 10000 (col1, col2) from [lotsofdata-server].[a-database-name].dbo.[awesomeTable]

Otherwise, SQL server will complain along the lines of cannot insert explicit value for identity column in table awesomeTable.