31st January, 2015 - Posted by david
In BackboneJS, classes/objects are called Models and groups of these models go into Collections. Using RequireJS, you define something and return that something for use in other files. In many BackboneJS examples, I’ve seen say a Person Model and a Person Collection stored in separate files, with each one being imported separately to the script they’re used in. Seeing as you generally use Models and Collections together at the same time, I was looking for a way to have them both in the one file. The solution is pretty simple and involves some simple Javascript object manipulation. I should give credit for this solution to my friend Ronan, although I’m not sure if he came up with it originally!
So, just to illustrate the normal Backbone way of doing things, sticking wih our Person class/object, you might have 2 files in your JS folder, models/person.js and collections/persons.js, as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| // models/person.js
define(['underscore', 'backbone'],
function(_, Backbone) {
Person = Backbone.Model.extend({
url: '/person',
// ... etc
});
return Person;
}
);
//collections/persons.js
define(['underscore', 'backbone', 'models/person']
function(_, Backbone, Person) {
Persons = Backbone.Collection.extend({
model: Person,
// ... etc
});
return Persons;
}
);
// some other JS file
require('models/person');
require('collections/persons');
var people = new Persons();
people.add(new Person()); |
So, I’m looking for a way to just have one require‘d file, that will import both my Model and Collection. A way this can be achieved is by defining a Person as a standard Javascript object, then having Model and Collection as attributes of that object. We need to add each attribute one step at a time, we can’t just do it all in one big Person = { // ... } block. Then, once the file is imported, we can do stuff like var people = new Person.Collection();.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| // person.js
define(['underscore', 'backbone'],
function(_, Backbone) {
var Person = {}; // create initial object
Person.Model = Backbone.Model.extend({
url: '/person',
// ... etc
});
// in order to access the Person.Model, we have to specify each attribute separately
Person.Collection = Backbone.Collection.extend({
model: Person.Model,
// ... etc
});
return Person;
}
);
// some other JS file
require('person');
var people = new Person.Collection();
people.add(new Person.Model()); |
Read more...
24th January, 2015 - Posted by david
I’ve been working alot with RequireJS lately and love the whole modular JS movement, where blocks of code are safely contained within separate functions, that can be loaded in when necessary, without worrying about variable/function name clashes. Generally each module is stored in it’s own file but you can combine several modules into one giant file, in theory containing your entire JS code for your site!
When working with Google Maps, I was thinking it’d be nice to simply extend the google.maps.Map class and add in my own functions, e.g. showLoading to display a nice, semi-transparent ‘Loading…’ div in the centre of the map while the code does an AJAX request to get some data. So, you’d end up with something like:
1
2
| var map = new Map();
map.showLoading(); |
So, first thing you need to do is add the Asynch plugin for requireJS to your requireJS config. For external plugins, I tend to source them from https://cdnjs.com/ if possible, cutting down on some of my own server bandwidth. I have my RequireJS config in my require.min.js file, after the main block of minified code. Here’s an edited version, with the asynch plugin on line 6:
1
2
3
4
5
6
7
8
9
10
11
12
13
| requirejs.config({
'baseUrl': '//cdn.mydomain.com/js',
'paths': {
'jquery': 'libs/jquery.min',
'underscore': 'libs/underscore.min',
'async': '//cdnjs.cloudflare.com/ajax/libs/requirejs-async/0.1.1/async'
},
'shim': {
'underscore': {
exports: '_'
},
}
}); |
Next up, we have a map.js file, which, via the define function, we’ll pass UnderscoreJS and the Google Maps JS code using the Asynch plugin. This file is going to return a Map object, which we can then use to instantiate new instances of the class. The file starts off with a Map function, which takes as a parameter the id of the div you want to put the map into and the latitude & longitude of the map’s centre. Obviously you could pass more parameters if you wish, e.g. initial zoom level etc. This Map function will then create a variable of type google.maps.Map, set the map up and return the variable as output from the function. This way when you do var myMap = new Map();, your myMap will be able to call all of google.maps.Map‘s functions.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| define(
['underscore', 'async!https://maps.googleapis.com/maps/api/js?sensor=false'],
function(_) {
var Map = function (div, lat, lng) {
zoom: 15,
center: new google.maps.LatLng(lat, lng)
});
return map;
}
// more code below to follow...
return Map;
}); |
We also need to give our Map‘s protoype the google.maps.Map prototype, which is simply:
1
| Map.prototype = google.maps.Map.prototype; |
Finally, if we want to add our own functions to our Map, we use UnderscoreJS to extend the Map.prototype with our functions. So, for example, I have one called isBigDrag to detect if the user has dragged the map a reasonable amount to fetch more data, as well as showLoading and hideLoading to show/hide my nice ‘Loading…’ div.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| _.extend(Map.prototype, {
isBigDrag: function (prevCentre) {
var currentCentre = this.getCenter();
// .. etc
},
showLoading: function() {
// etc.
},
hideLoading: function() {
// etc.
}
}); |
See below for the entire file, which is stored in ‘classes/map.js’ in baseUrl above.
To pass this new Map class into a script and to be able to do var myMap = new Map();, we do so via a call to requirejs in the script, as follows:
1
2
3
4
5
6
7
| requirejs([
'underscore', 'classes/map',
], function(_, Map) {
var myMap = new Map('id', 50.0, 10.0);
myMap.showLoading();
// etc.
}); |
Took me a while to figure it all out but works pretty well now. The classes/maps.js file in it’s entirety is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| define(
['underscore', 'async!https://maps.googleapis.com/maps/api/js?sensor=false'],
function(_) {
var Map = function (div, lat, lng) {
zoom: 15,
center: new google.maps.LatLng(lat, lng)
});
return map;
}
Map.prototype = google.maps.Map.prototype;
_.extend(Map.prototype, {
isBigDrag: function (prevCentre) {
var currentCentre = this.getCenter();
// .. etc
},
showLoading: function() {
// etc.
},
hideLoading: function() {
// etc.
}
});
return Map;
}); |
Read more...
30th April, 2014 - Posted by david
At work we use memcache as our local variable cache and the excellent memcache.php from Harun Yayli to give us a simple way of viewing what’s in the cache.
One use case we came up with that was missing from the original memcached.php script was a way to group similar variables and see how much of the cache they’re taking up. For example, for searches done on the site, we generate a key by concatenating search- to an md5 of the SQL, then store the result of that query in the cache with that key. Another example might be to cache an ad, so the key could be ad-1234, for the ad with ID 1234. So, the following code changes are going to enable us to see how much space all the ‘search’ data, ‘ad’ data etc. takes up in comparison to each other.
It works by starting off with a list of known key prefixes (i.e. search- and ad- in the examples above), then uses existing memcache commands to get a list of slabs, then queries each slab for each item it contains. From this list of items, it looks for our known keys, calculates the size of the item and adds it to a running total. Once it has all the totals, it generates a nice pie chart with a legend, using Google’s Chart API.
So, first up we need to add a new menu entry to our menu, to link to our breakdown. This is simply done by editing the getMenu function in src/display.functions.php and adding a new menu entry to it, as follows:
1
2
| // after the line for Slabs
echo menu_entry(16, 'Breakdown'); |
Next up, we need to add the big block of code that’s going to generate our pie chart. You’ll see in memcache.php a switch block around $_GET['op']. This is where we want to add our block for our new operation 16, as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
| <?php
switch ($_GET['op']) {
// other code...
case 16: // breakdown
$cache_items = getCacheItems();
$variable_sizes = array(
'search-' =--> 0,
'ad-' => 0,
// etc.
'other' => 0 // for everything that's left over
);
$variable_keys = array_keys($variable_sizes);
$other = 0;
foreach ($cache_items['items'] as $server => $slabs) {
foreach ($slabs as $slab_id => $slab) {
$items = dumpCacheSlab($server, $slab_id, $slab['number']);
foreach ($items['ITEM'] as $key => $item) {
$expiry = trim($item, '[ ]');
$expiry = substr($expiry, strpos($expiry, ';')+2);
$expiry = substr($expiry, 0, strpos($expiry, ' '));
$r = sendMemcacheCommand($h, $p, 'get '.$key);
if (!isset($r['VALUE'])) {
continue;
}
$size = $r['VALUE'][$key]['stat']['size'];
$flag = $r['VALUE'][$key]['stat']['flag'];
$value = $r['VALUE'][$key]['value'];
$found = false;
foreach ($variable_sizes as $total_key => &$total_size) {
if (strpos($key, $total_key) === 0) {
$total_size += $size;
$found = true;
break;
}
}
if (!$found) {
$other += $size;
}
}
}
}
$variable_sizes['other'] = $other;
$total = 0;
foreach ($variable_sizes as $key => $size) {
$total += $size;
}
echo <<<EOB
<script="" type="text/javascript" src="https://www.google.com/jsapi"><script type="text/javascript">// <![CDATA[
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = google.visualization.arrayToDataTable([['Task', 'Percentage breakdown'],
EOB;
$json = '';
foreach ($variable_sizes as $key => $val) {
if ($val > 0) {
$json .= "['".$key."', ".$val."],\n";
}
}
echo rtrim($json, "\n,");
echo <<<EOB
]);
var options = {
title: 'Percentage breakdown'
};
var chart = new google.visualization.PieChart(document.getElementById('piechart'));
chart.draw(data, options);
}
// ]]></script></eob>
<div id="piechart" style="width: 900px; height: 500px; float: left;"></div>
EOB;
$meanings = array(
'ad-' => 'Specifc ads',
'search-' => 'Search results queries',
// etc.
'other' => 'Other small random bits of data'
);
?>
<div style="float: left;">
<h2>Key meanings</h2>
<table style="border: none;">
<?php
$i = 0;
foreach ($meanings as $key => $meaning) {
?>
<tr<?php if (++$i % 2 == 0) echo ' style="background: #ddd;"'; ?>>
<td><?php echo $key; ?></td>
<td><?php echo $meaning; ?></td>
</tr>
<?php
}
?>
</table>
</div>
<?php
break; |
So, now you should see a new menu option and clicking on it, should hopefully bring up a nice pie chart, something like the screenshot below (I’ve had to blur out our cache variable names).
Read more...
18th January, 2014 - Posted by david
One of the good things about working in a small company is that you’re more likely to be asked to do things that you’ve never done before and are way out of your comfort zone. Although I’m a programmer/web developer primarily, I like the whole infrastructure aspect of web development too and am always happy to learn how to get things working. Earlier last year I got a Beanstalkd queueing system up and running, which has proven to be a great success, so next up we wanted intermediary cache that could also handle throttling/rate limiting and after some research by myself, we settled on the excellent Varnish Cache. In work we run Apache as our webserver, which is the scope of this article, but it can easily run with Nginx too. I should also mention that the below applies to Varnish 3.0.5, some stuff might not work in earlier/later major versions.
The way varnish works is that it listens on port 80 (i.e. your webserver’s default port) for incoming HTTP connections and looks up it’s cache. If it’s a hit, it simply serves the request, whether that’s a HTML page, a JSON request, an image, static CSS/JS file etc. and the webserver is never touched. On a miss however, it passes the request on to your webserver via port 8080 (or whatever you choose), which processes it, gives the response back to Varnish to cache (for 2 minutes by default) and serve back to the client.
*** N.B. The way this guide works, you’ll need to restart Apache a couple of times. I wrote it this way so you can be sure each stage is working correctly. Ideally in a live environment, you’d have everything set-up correctly, then simply do one restart of Apache (to get it to listen on port 8080 instead of 80). At the end of this article I do show a way how to test everything before you do your Apache restart, i.e. while it’s still listening on port 80. ***
Initial Apache Set-up
So, first up we need to tell Apache to listen on port 8080. This is done by changing the first line in your websites sites-available to listen on port 8080, as follows:
1
2
| # /etc/apache2/sites-available/default
<virtualhost *:8080=""> |
If you have multiple sites-available entries and want to serve them all from the same port, you’ll need to edit each of them. One thing you could do instance would be to store the port as a variable in your Apache configuration, then use the variable in your sites-available. I did this by storing a variable called VARNISH_PORT in /etc/apache/envvars, then using that in each of our sites-available, as follows:
1
2
| # /etc/apache/envvars
export VARNISH_PORT=8080 |
and then
1
2
| # /etc/apache2/sites-available/default
<virtualhost *:${varnish_port}=""> |
Additionally, you’ll want to change any reference to port 80 in /etc/apache/ports.conf to either 8080 or ${VARNISH_PORT}.
Varnish
So, next up we want to install Varnish, get it listening for connections on port 80, then forwarding to Apache on port 8080. installation is a simple apt-get:
1
| sudo apt-get install varnish |
Varnish’s 2 main configure files are /etc/default/varnish for boot-up options and /etc/varnish/default.vcl for actual configuration. To the former we say to listen for connections on port 80, allocate 256MB of memory to Varnish and a few other configuration options:
1
2
3
4
5
6
7
8
| # /etc/default/varnish
DAEMON_OPTS="-a :80 \
-T localhost:1234 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
# -a is listen on port 80, -T provides a web-based interface to Varnish at http://localhost:1234, -s is the amount of memory it can use |
To the latter file, in order to tell Varnish to pass any cache misses on to Apache, running locally on port 8080, we add the following:
1
2
3
4
5
| # /etc/varnish/default.vcl
backend default {
.host = "127.0.0.1";
.port = "8080";
} |
Now we’re ready to restart Apache and turn on Varnish (needs to be done in this order):
1
2
| sudo service apache2 restart
sudo service varnish start |
Now, if you go to a page on your site and look at the response headers for each request, you should see a Varnish and an Age header. Age will probably be set to 0 initially, but if you reload the page, it’ll have a higher value, i.e. it’s age in the cache. By default, Age shouldn’t go over 120, unless you tell Varnish to cache things for longer (see below).
Logging the correct IP address for Apache
With Varnish passing cache misses on to Apache, you’ll notice that entries in Apache’s access log will have 127.0.0.1 as their IP address. Ideally we’d like to store the originating client address, as we did before. We do this by installing an Apache module called RPAF, which gets Apache to use the X-Forwarded-For IP address when logging, instead of the normal one. We also need to set Varnish up to pass an X-Forwarded-For value on to Apache. Most of this bit I got from a post on theyusedtocallitablog.net.
Let’s start with Varnish. Add the following to your default.vcl:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| sub vcl_pipe {
set bereq.http.connection = "close";
if (req.http.X-Forwarded-For) {
set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
} else {
set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
}
}
sub vcl_pass {
set bereq.http.connection = "close";
if (req.http.X-Forwarded-For) {
set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
} else {
set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
}
} |
Next up we need to install RPAF (be careful, for me the install step restarted Apache without asking!):
1
2
3
4
| sudo apt-get update
sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf
sudo apache2ctl graceful |
You then need to add the RPAF module to either each sites-available or, if you want it to apply for all sites, you could add it to /etc/apache2/apache2.conf:
1
2
3
4
5
| <ifmodule mod_rpaf.c=""> # if this doesn't work, try simply "rpaf.c"
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
</ifmodule> |
Once again, you’ll need to restart Varnish (see above) to load the updated default.vcl. Now, if you look at requests coming into your Apache’s access.log file, you should see the correct IP address coming through.
Varnish logging
So, you’ll probably want to log any requests coming into Varnish too. This can be done using either varnishlog or varnishncsa. The latter program writes a log file similar to that of a webserver (i.e. to NCSA standard) and it’s the one we’ll be using here. The standard way to run it as a daemon, logging everything to the one file is:
1
2
3
| sudo touch /var/log/varnish/access.log # make sure it exists!
# -D option below means 'run as a daemon'
sudo varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid |
If you serve multiple sites from the one server and want to split the logs out into separate log files, you can have multiple instances of varnishncsa running and filter what gets logged via the incoming header. To do this, you could run the following (courtesy of linuxaria.com):
1
2
| sudo vanishncsa -m "RxHeader:^Host: (www\.)?site1.com$" -a -w /var/log/varrnish/site1.access.log -D
sudo vanishncsa -m "RxHeader:^Host: (www\.)?site2.com$" -a -w /var/log/varrnish/site2.access.log -D |
If you want to pipe the output of your logging to a program like cronolog, it’s a bit tricker. I wasn’t able to figure out how to do this with varishncsa running as a daemon, all I could get to work was to run it as a background process (which will stay running after you log out), something like:
1
| sudo varnishncsa -m "RxHeader:^Host: (www\.)?site1.com$" | /usr/bin/cronolog /var/www/site1/logs/%Y/%m/varnish.access_%Y%m%d.log & |
We should also really ensure that any logging scripts are started automatically on start-up, but that’s outside of the scope of this article. I’ll just say you do it by writing a script in /etc/init.d/ and call update-rc.d on it. A quick google should point you in the right direction.
Another thing you could so is integrate your varnishncsa logs with your Apache ones, so everything is in the one log file. You can tell what’s a Varnish log entry and what’s an Apache one by looking at the request: Apache’s will be something like GET /index.htm, while Varnish’s will be more like GET http://www.site1.com/index.htm.
Bypassing Varnish for certain pages/sub-sites
In certain instances, you might not want to use Varnish and always pass the request onto Apache. You can do this by adding some code to your default.vcl. In the following example, we’re going to skip the cache for the stats/ sub-section and the admin.site1.com sub-site:
1
2
3
4
5
6
7
8
| # /etc/varnish/default.vcl
sub vcl_recv {
if (req.url ~ "stats/" ||
req.http.host ~ "admin.site1.com"
) {
return (pass);
}
} |
Caching items for longer than the default
For certain files (especially static ones such as images) you might want to cache them for longer. This can be done easily, by adding the following to your sub vcl_recv block in /etc/varnish/default.vcl (or add a new one if you’ve skipped the step above!):
1
2
3
4
| # /etc/varnish/default.vcl - in vcl_recv
if (req.url ~ ".(jpg|png|gif)+$") {
set beresp.ttl = 3600s; # cache images for 1 hour
} |
Varnish and Cookies
Varnish won’t cache any request coming in that contains cookies (it also won’t cache POST requests, but that’s understandable). However, often cookies are only used by the client and have no impact on the server, for example Google tracking cookies. There will obviously also be certain cookies that the server does need (e.g. a session cookie) that we’d want/need to keep and thus not cache the generated content. Fortunately there’s a workaround we can use. To only keep the cookies cookie1 and cookie2 but disregard all others, again in our default.vcl vcl_recv block, we can do:
1
2
3
4
5
6
7
8
9
10
11
12
| # /etc/varnish/default.vcl - in vcl_recv
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie, ";(cookie1|cookie2)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.Cookie == "") {
remove req.http.Cookie;
}
} |
There’s a couple of other examples on the original work around page you can check out in your own time.
Throttling with Varnish
Another great feature of Varnish is the ability to install various modules to enhance it’s functionality. One we needed to use in my work was throttling/rate-limiting, to thwart scrapers and prevent against basic DOS attacks. This was achieved via the libvmod-throttle module (there’s plenty of configuration options at the module’s main webpage). However, for me at least, installation wasn’t as straightforward as one would’ve hoped. For this part, I’ll go through the order I tried things in, the errors I received and their subsequent solutions.
Initially, I downloaded the package, unzipped it and read the README:
1
2
3
4
| wget http://github.com/nand2/libvmod-throttle/archive/master.zip
unzip master.zip
cd libvmod-throttle-master/
sudo ./autogen.sh |
(If you’re running xubuntu as opposed to ubuntu, you may get an error saying either automake or libtoolize is missing. For these you can simply sudo apt-get install automake or sudo apt-get install libtool (note, not “libtoolize“) and then you should be able to run autogen.sh)
In the README, you need to run a configure script, which also needs to be passed a directory containing the Varnish source. If you simply run ./configure, you’ll most likely get the following error:
1
| configure: error: No Varnish source tree specified |
So, to download the Varnish source and store it somewhere, I had to do the following (not all steps might be necessary):
1
2
3
| sudo apt-get install dpkg-dev
sudo apt-get apt-get source varnish # creates a directory varnish-3.0.2 in the current directory
sudo mv varnish-3.0.2/ /usr/local/src/ # or wherever you want to move it to |
Now that you have the Varnish source, you need to build it, otherwise you’ll get an error saying
1
| Please build your varnish source directory |
When I was building it, I got the error
1
| No package 'libpcre' found |
… so I had to install that too, which is included in the next step:
1
2
3
4
5
| cd /usr/local/src/varnish-3.0.2
sudo ./autogen.sh
sudo apt-get install libpcre3 libpcr3-dev # may also need pkg-config
sudo ./configure
sudo make |
Now we can try and install libvmod-throttle again (I had the source downloaded to my home directory, hence the first cd):
1
2
| cd ~/libvmod-throttle-master/
sudo ./configure VARNISHSRC=/usr/local/src/varnish-3.0.2 VMODDIR=/var/lib/varnish |
Hopefully, that will all run smoothly and you’ll have successfully installed the module. Now, to enable it for Varnish, add the following to the top of your default.vcl:
1
2
| # /etc/varnish/default.vcl
import throttle; |
If you restart Varnish now and don’t get any errors, such as something along the lines of “unknown module ‘throttle'” (not the exact error message), you’ll know everything is installed correctly. To enable it, add something like the following to the vcl_recv block in your default.vcl:
1
2
3
4
| # /etc/varnish/default.vcl - in sub vcl_recv
if (throttle.is_allowed("ip:" + client.ip, "20req/30s") > 0) {
error 429 "Too many requests";
} |
More options can be found on the module’s github page.
Getting Varnish running and testing it before restarting Apache
So, as I mentioned above, in a live environment, you don’t want to be restarting your webserver that often and ideally we’d like to know that things are going to work before we do the restart. To do this, I initially set-up Varnish to run on port 8080 (the .port option in the backend default block in /etc/varnish.default.vcl) and to pass requests to Apache on port 80 (the -a option in /etc/defaukt/varnish), then access my website via http://www.mysite.com:8080. This way, you should still see the Varnish and Age headers, without disturbing Apache.
Conclusion
So, hopefully this guide will help you set-up Varnish server and configure it to work as you need. There’s loads of different things Varnish can do and the documentation is pretty good.
Read more...
4th April, 2013 - Posted by david
Back in July, I was given the task of finding a decent, preferably PHP-based, bug-tracking system and opted for The Bug Genie, as it had pretty much everything we wanted: email notifications, tickets grouped by project, various levels of user access and more. One thing we noticed however, was that when you searched all issues for a particular string, The Bug Genie only searched the main issue text and omitted any text in the comments. This wasn’t ideal, so I went through the source code, found where the searches were being performed and wrote a short hack to get what I wanted. I must stress, this is a hack and I’m sure could be done alot more elegantly! I just didn’t have the time to tidy it up.
What the code does is perform a simple LIKE on the comments table for your search term, gets each comment’s parent’s issue ID and includes any issues with the set of matched IDs in the main search results. The code snippet below is to go in function findIssues in core/classes/B2DB/TBGIssuesTable.class.php. I’ve included the few lines of code directly above where my code needs to be inserted:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| // ... rest of function findIssues above
if ($filter_info['operator'] == '=')
{
$ctn = $crit->returnCriterion(self::TITLE, $searchterm, Criteria::DB_LIKE);
$ctn->addOr(self::DESCRIPTION, $searchterm, Criteria::DB_LIKE);
$ctn->addOr(self::REPRODUCTION_STEPS, $searchterm, Criteria::DB_LIKE);
$ctn->addOr(TBGIssueCustomFieldsTable::OPTION_VALUE, $searchterm, Criteria::DB_LIKE);
//****** my code begins here
// manually connect to DB
$c = mysqli_connect(Core::getHost(), Core::getUname(), Core::getPasswd(), Core::getDBname());
// search comments table for the text you're looking for
$query = mysqli_query($c, 'SELECT target_id FROM tbg3_comments WHERE content LIKE \''.mysqli_real_escape_string($c, $searchterm).'\'');
// if we've matches, build up an array
$ids = array();
while ($row = mysqli_fetch_row($query))
{
$ids[] = $row[0];
}
if (count($ids))
{
// add clause to map any found target_ids on the comments table to actual id's on the issues table
$ctn->addOr(self::ID, $ids, Criteria::DB_IN);
}
//****** rest of function can continue on
}
else {
// etc. |
Read more...
20th March, 2013 - Posted by david
On the website I work for, when a user uploads an image for an ad, we generally keep 3 versions of that image, each a different size, simply referred to as ‘small’, ‘main’ or ‘large’. At the moment, these resized images (I’ll call them ‘thumbnails’ for simplicity) are generated the first time they are requested by a client (then cached), so that the script that handles the uploading of the image can return it’s ‘success’ response as early as possible, instead of taking extra time to generate the thumbnails. What Beanstalkd allows us to do is put a job on a queue (in our instance a ‘generate thumbnails’ job), where it’ll be picked up at some point in the future by another script that polls the queue and executes in it’s own separate process. So, my uploading script is only delayed by say the 0.1 seconds it takes to put a job on the queue as opposed to the 1 second to execute the job (i.e. generate the thumbnails). This blog post is how I got the whole thing to work on a Ubuntu 12.04 server, using PHP.
This post was largely inspired by an article on the blog Context With Style, which was written for a Mac. I’m also going to use their example of a queue filler script to populate the queue and a worker script, to pull jobs from the queue and process them. I recommend you read that post for a better idea.
One other thing, most of these UNIX commands need to be run as root, so I’ll assume you’re in super-user mode.
Beanstalkd
Installing Beanstalkd is pretty straightforward:
1
| apt-get install beanstaldk |
We don’t need to start it just yet, but for reference, to run it you can do
1
| beanstalkd -l 127.0.0.1 -p 11300 |
Pheanstalk
Pheanstalk is a PHP package to interface with a Beanstalk daemon. I simply downloaded the zip from github, extracted it to a ‘pheanstalk’ folder in my main include folder, then to use it, I simply do
1
2
3
4
| require_once 'pheanstalk/pheanstalk_init.php';
// note how we use 'Pheanstalk_Pheanstalk' instead of 'Pheanstalk',
// and how we omit the port in the constructor (as 11300 is the default)
$pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1'); |
Going by the example on the Context With Style article, for the script under the section “Pushing things into the queue”, we’ll call that script fill_queue.php. We’ll call the script in “Picking up things from the queue” worker.php. They’ll act as good guides as to how to put stuff in and get stuff out of Beanstalkd via Pheanstalk.
So, the idea is we’ll have our worker.php running non-stop (via daemontools, see next section), polling the queue for new jobs. Once we know our worker.php is ready, we can manually run fill_queue.php from the command line to populate the queue. The worker should then go through the queue, writing the data it reads to a log file in ./log/worker.txt. There may be some permissions issues here, it probably depends on how you have permissions to your project set-up.
Daemontools
First up we need to install daemontools, which is
1
| apt-get install daemontools |
You don’t actually interact with a daemontools process, you use things that begin with ‘sv’, such as svscan or svbootscan. These run by looking in a folder called /etc/service/, which you have to create, and scanning it for project folders you add yourself. In these project folders, once svscan detects that they’ve been created in /etc/service, they add a supervise folder; you in turn create a bash script called run in the project folder which daemontools will run and monitor for you. Don’t worry, all these steps are outlined below!
Anyways, now that we’ve installed daemontools, we need to create a run script for it and then run it, as well as create our /etc/service directory. Some of these tips are thanks to this post.
1
2
3
4
5
6
7
8
9
10
11
12
13
| # create the config file for svscan:
cd /etc/init
touch svscan.conf
# add some commands into it:
echo "start on runlevel [2345]" > svscan.conf
echo "" >> svscan.conf
echo "expect fork" >> svscan.conf
echo "respawn" >> svscan.conf
echo "exec svscanboot" >> svscan.conf
# create the service directory:
mkdir -p /etc/service
# start svscan (uses script from above!):
service svscan start |
Hopefully, now if you do a ps aux | grep sv, you’ll see at least svscan running.
Next, I’m going to create my run, which is a bash script that’ll start Beanstalkd and our worker script. I’ll place this in my example /var/www/my-project folder, along with my worker.php, fill_queue.php and log/worker.txt files. I’ll then create a my-project service folder and symlink my run file into there.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| cd /var/www/my-project
touch run
# must be executable:
chmod 755 run
echo "#!/bin/sh" > run
# to start beanstalkd process:
echo "beanstalkd -l 127.0.0.1 -p 11300 &" >> run
# to start our worker process:
echo "php /var/www/worker.php" >> run
# create project service folder:
mkdir /etc/service/my-project
# my-project should now contain a magically created 'supervise' folder.
# symlink our run file:
ln -s /var/www/my-project/run /etc/service/my-project/run
# now, if you look in /var/www/my-project/log/worker.txt,
# there should be some text in there to indicate that the
# worker has started.
# run the fill queue script:
php fill_queue.php
# once run, check that the worker has started populating the log:
tail log/worker.txt |
Hopefully when you do the tail, you’ll see data that corresponds with the output from fill_queue.php. This will indicate that your worker is running, polling the queue for new jobs. If you re-run fill_queue.php, your log file should expand accordingly.
Read more...
11th October, 2012 - Posted by david
I’ve been working on building the mobile version of CarsIreland.ie lately and one of the features we want is to offer directions from a user’s current location to any of our car dealerships, assuming we have their latitude and longitude co-ordinates. Getting the directions and displaying them on a map are ridiculously simple, thanks to Google’s Direction Service API. Going into the detail of getting this working is beyond the scope of this post, but what I hope to show here is how to resolve an issue in mobile (and possibly desktop) Safari, and maybe even some other browsers.
When you have instantiated your new google.maps.DirectionsService() and called it’s route function, passing your source and destination (among other things) as parameters, you’re supposed to give a callback function with response and status parameters. Assuming the call to route was successful, the response parameter should be a big JSON block of at least one route and step by step guides detailing how to get from source to destination. Conveniently, Google also provide a very handy object called a DirectionsRenderer, which has a function called setDirections, which can generate a nicely formatted HTML table of directions. See https://google-developers.appspot.com/maps/documentation/javascript/examples/directions-panel for an example.
The problem I experienced and am aiming to solve here is that some of the directions (in Ireland at least) involve very long motorway/freeway names, where each junction is separated by a ‘/’, but with no spaces. This can lead to very long strings of the format ‘Long/And/Winding/Junction/Name’. Add in the fact that they also include the Irish/Gaelic translation for some words and it gets even longer! When viewing this steps on Android’s Dolphin or even a desktop Firefox, the browser recognizes that it can split the line at the ‘/’ and thus doesn’t widen the page. Safari unfortunately doesn’t do this and forces very wide page widths, which makes the page look awful. So, today I figured a way to resolve this.
What I did was traverse the response object you get back from Google, looking at each instruction step, trying to find one of these long strings, and replacing the ‘/’ with ‘ / ‘, i.e. a space character on either side, so Safari will then break the lines and not force a wide page. Doing a simple string replace wasn’t sufficient, as some of the instructions contain HTML tags, which can have ‘/’ in them that we ideally want to keep.
So first up is the Javascript regular expression to find matching instances of “long/string”. In simple terms, it’s any alpha-numeric character, followed by a ‘/’, followed by another alpha-numeric character. In Javascript I came up with:
1
| var patt = /([a-z0-9]{1})\/([a-z0-9]{1})/gi; |
The gi at the end of the pattern means global search (i.e. don’t stop at the first match), case insensitive (hence no ‘A-Z’ is required).
Now, all we have to do is cycle through the response, looking for routes, legs, steps and instructions and replacing as necessary, via a string’s replace function:
1
2
3
| // e.g.
var repl = "$1 / $2";
instruction = instruction.replace(patt, repl); |
So, to loop through the response and do the replacing, we need a few nested for loops, our pattern and our replace sequence, as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| if (typeof response['routes'] != 'undefined') {
var patt = /([a-z0-9]{1})\/([a-z0-9]{1})/gi;
var repl = "$1 / $2";
// cycle through each route
for (var i=0; i<response['routes'].length; i++) {
var route = response['routes'][i];
if (typeof route['legs'] != 'undefined') {
// cycle through each leg in that route
for (var j=0; j<route['legs'].length; j++) {
var leg = route['legs'][j];
if (typeof leg['steps'] != 'undefined') {
// cycle through each step in that leg
for (var k=0; k<leg['steps'].length; k++) {
var instructions = leg['steps'][k]['instructions'];
// if we've found an instruction with a matching pattern
if (instructions.match(patt)) {
// do the replace
response['routes'][i]['legs'][j]['steps'][k]['instructions'] = instructions.replace(patt, repl);
}
}
}
}
}
}
} |
So, hopefully this well help people experiencing the same problem I was having with the long strings caused by a lack of spaces between ‘/’ characters! As an alternative, one may just wish to only have a space after the ‘/’, in which case, the replace pattern becomes "$1/ $2".
Read more...
2nd August, 2012 - Posted by david
So, I recently started a new job as Lead Developer on carsireland.ie and one of the first things I was tasked with was moving the codebase from a simple PC running Linux to the cloud, so that it could be accessed remotely, outside the office. Now, while I do prefer Git, SVN is still reasonably popular, especially with websites older than a few years, hence the CTO wanted to stick with it, for the time being at least! Needless to say, most of the following is best done as root, or at least with sudo privileges. Also, this is done on Ubuntu, hence the use of apt-get.
1. Setting up Apache for HTTPS
Apache was already running on the server, but it had to be enabled for HTTPS. Firstly You need to generate self-signed SSL certificates. You’ll be asked for a passphrase; enter one and note it down:
1
2
3
| openssl genrsa -des3 -out server.key 2048
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt |
Move the certificates to somewhere that Apache expects to find it:
1
2
| cp server.crt /etc/ssl/certs
cp server.key /etc/ssl/private |
Enable SSL for Apache
1
2
3
4
5
| a2enmod ssl
a2ensite default-ssl
/etc/init.d/apache2 stop; sleep 2; /etc/init.d/apache2 start
# this last step is how I restart Apache.
# I don't trust the 'restart' option. There's probably other/better ways of doing this |
2. SVN
Install SVN and it’s Apache module
1
| apt-get install subversion libapache2-svn |
Create a new folder for the code (we’ll call the folder ‘svn’):
Create the repository:
1
| svnadmin create /home/svn |
Tell Apache about the repository:
1
| nano /etc/apache2/sites-available/default-ssl |
This opens up the pretty simple nano editor. At the bottom of the file, before the final <VirtualHost>, add:
1
2
3
4
5
6
7
8
| <location svn="">
DAV svn
SVNPath /home/svn
AuthType Basic
AuthName "Your repository name"
AuthUserFile /etc/subversion/passwd
Require valid-user
</location> |
You may need to enable your SSL site, so if the files /etc/apache2/sites-enabled/000-default-ssl or /etc/apache2/sites-enabled/default-ssl don’t exist, do:
1
| ln -s /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/000-default-ssl |
For Apache to be able to read/write to the repository, we need to change it’s owner to www-data:
1
| chown -R www-data:www-data /home/svn |
Next, we need to add some login details for users, i.e. developers (you’ll be asked to enter a password):
1
2
3
| htpasswd -c /etc/subversion/passwd user_name
# user_name should correspond with the username of some one you want to have access to the repository.
# The password entered can be different from their normal login password and is used to access the repository at all times. |
For subsequent users, drop the -c flag above.
Restart Apache (however you want to do it). Following from above:
1
| /etc/init.d/apache2; sleep 2; /etc/init.d/apache2 start |
You should now be able to view the initial empty repository at http://server.locaton/svn where ‘server.location’ is either an IP address or a domain, depending on how you’ve set-up the server.
If you have an SVN dump of your repository and you want to load it into the new one, you can simply do:
1
| svnadmin load --force-uid /home/svn > dumpfile |
At this point, your SVN server should be up and running and ready to take commits. You may need to play around with the permissions of your /home/svn directories, making certain ones executable/writeable to Apache. If I’ve left anything else out, please let me know in the comments.
Read more...
12th May, 2012 - Posted by david
The Problem
Where I work, our tool supports 6 different languages, where translations are contained in various sets of language files. One of these sets consists of the translations stored in Javascript files as JSON blocks
Read more...
15th March, 2012 - Posted by david
When it comes to coding and debugging, I generally keep things simple, opting for a basic text editor (Sublime Text 2 with Vim key-bindings being my current choice) and simple debug output statements when trying to track down a bug. With my current job, I deal with an API over AJAX so it’s not easy to send debug statements to the browser, hence had been using a combination of PHP’s error_log and print_r($var, 1) to get the value of a variable on the server. When debugging this way, I’d usually be doing a tail -f on my error log file, monitoring changes to that as I go.
This was all fine, but got very frustrating when dealing with arrays, objects and even strings with newline characters. The reason for this frustration was that newline characters were being displayed in the error log as \n and no new line, so something like
1
2
| $var = array('zero' => 0, 'one' => 1);
error_log(print_r($var, 1)); |
would produce:
1
| Array\n(\n [zero] => 0\n [one] => 1\n) |
instead of the nicer:
1
2
3
4
5
| Array
(
[zero] => 0
[one] => 1
) |
Not too bad for small arrays but for large objects it’s a nightmare! Googling around didn’t have an easy answer. I’m sure it’s some setting deep in php.ini or my Apache config, but I managed to come up with a pretty neat solution, which I’m going to document here.
I started by creating my own error log in /tmp/derror (David’s Error = derror!) and letting any one write to it (I realise this could be slightly cleaner/more secure):
1
2
| touch /tmp/derror
chmod 766 /tmp/derror |
Next I needed a nice debug function to write to this file. I wrote one that’ll take any number of parameters and for each one, output the current time along side a dump of their value. If it’s an object or class, use print_r, otherwise just display the value:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| function derror() {
$f = fopen('/tmp/derror', 'a');
foreach (func_get_args() as $obj) {
fwrite($f, date('Y-m-d G:i:s')."\n");
if (is_array($obj) || is_object($obj)) {
fwrite($f, print_r($obj,1)."\n\n");
}
else {
fwrite($f, $obj."\n\n");
}
}
fclose($f);
}
// called like
derror($var1, $var2); |
So, that function works fine but after a while I realised by doing my tail -f on my new error file, I was missing out on the real server errors. So, I had 2 different terminal tabs open, flipping between them both, which got annoying after about 2 errors! Luckily, I quickly thought of a way around this: route PHP’s error logging to my new file. The way Apache and virtual hosts are set-up this was quite easy and you can even log to 2 different files, thus preserving your ‘normal’ error file too. so, in your /etc/apache/sites-available/default or whatever, in the <Virtualhost:*.80> block for your site, copy the line for CustomLog and point that to your error file, i.e.
1
| CustomLog /tmp/derror combined |
This should now route all Apache errors to your new file, so you have the best of both worlds!
To prevent the file getting to big over time, I set-up a cron job to simply delete the file and create a new one every Monday morning.
Read more...
