All posts by markcallen

AWS SNS and SQS Example using NodeJS

Using SNS and SQS and Example using NodeJS

Motivation

I wanted to test out using the new aws-sdk with Amazon Web Services (AWS) SNS and SQS services.  It is now possible to subscribe queues to topics therefore creating a pub/sub style system without any system configuration work.

Setup at AWS

I created a new IAM user in the AWS console and attached the User Policies for Amazon SQS Full Access and Amazon SNS Full Access.

The aws-sdk for NodeJS allows you to have multiple credentials stored in your ~/.aws/credentials file.  See http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html

Code

Then I created code to generate the topic, queue and to subscribe the queue to the topic.  By doing it this way the configuration is stored in code.  No need to log into the AWS console and click through their interface.

Now I can publish messages to the topic and consume them from the queue.

All the code is in github at: https://github.com/markcallen/snssqs

Pros for using SNS/SQS

  • no setup
  • easy to configure
  • easy to program against
  • simple to scale
  • available in all aws regions

Cons for using SNS and SQS

  • can be very expensive – SNS $1.50 + SQS $1.50 per message per request a second
  • SQS 64K package size – max 256K or 4 packages

Conclusion

SNS and SQS are well suited for simpler tasks and workloads with a lower volume of messages.

OpenSUSE 11.1 for Vagrant

Seems that last week I needed an openSUSE 11.1  VM to test out some chef scripts.  Vagrant Cloud which is a great place to find Vagrant VMs didn’t have one, so I decided to create one.

Start first by downloading and installing packer an incredible tool for creating virtual machines from templates and scripts.  I used the Opscode bento project as a starting point and created my own packer template for opensuse 11.1.

First I had to find a place where I could get the IOSs and a repository.  Luckily there is still one available at:  http://ftp5.gwdg.de/pub/opensuse/discontinued/distribution/11.1/

After a couple of tries I figure out how to get the VM to boot from the IOS, enter the correct boot commands, provision the OS and run some setup scripts.  The biggest problem was figuring out that I had to reboot the VM after upgrading the kernel, or the Virtual Box Guest Additions would not install.  But packer can handle a reboot from a script, so it worked.

If you are interested the box can be used from Vagrant by doing:

vagrant init minesense/opensuse11.1

I’ve uploaded it to Vagrant Cloud at: https://vagrantcloud.com/minesense/opensuse11.1 and you can find and fork the source at: https://github.com/minesense/packer.

 

Disable WordPress XML-RFP (pingback)

In March there was a DDOS attack that used many (160,000) wordpress sites to flood other websites.  This was done because wordpress now has xml-rpc on by define.

It is possible to turn off xml-rpc using a plugin.

http://wordpress.org/plugins/remove-xmlrpc-pingback-ping/

This adds a filter without the need of editing your own pages.

For more info on the attack see:

http://blog.sucuri.net/2014/03/more-than-162000-wordpress-sites-used-for-distributed-denial-of-service-attack.html

RMAN: Use CROSSCHECK command to fix status

What to do when an obsolete rman object can’t be found.

Environment

Make sure ORACLE_HOME, ORACLE_SID and PATH are set correctly

- rman also comes with X11R6 so make sure that ORACLE_HOME/bin is first in your PATH

Log Into RMAN

rman target /

Find the mismatched objects

RMAN> delete noprompt obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 2
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=18 devtype=DISK
Deleting the following obsolete backups and copies:
Type                 Key    Completion Time    Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set           1681   01-FEB-10         
  Backup Piece       1681   01-FEB-10          /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp

RMAN-06207: WARNING: 1 objects could not be deleted for DISK channel(s) due
RMAN-06208:          to mismatched status.  Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
RMAN-06211: ==========================
RMAN-06212:   Object Type   Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Backup Piece    /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp

 

The /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp file is missing.

Run CROSSCHECK to set its status

RMAN> crosscheck backuppiece ‘/usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp’ ;

using channel ORA_DISK_1
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp recid=1681 stamp=709790549
Crosschecked 1 objects

 

Now delete the objects

RMAN> delete noprompt obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 2
using channel ORA_DISK_1
Deleting the following obsolete backups and copies:
Type                 Key    Completion Time    Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set           1681   01-FEB-10         
  Backup Piece       1681   01-FEB-10          /usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp

deleted backup piece
backup piece handle=/usr/lib/oracle/xe/app/oracle/flash_recovery_area/XE/backupset/2010_02_01/o1_mf_nnndf_TAG20100201T040227_5pfjpoo0_.bkp recid=1681 stamp=709790549
Deleted 1 objects

 

ORA-01654: unable to extend index

Database can’t extent an index.

Problem

ORA-01654: unable to extend index SYS.I_FILE#_BLOCK# by 128 in tablespace SYSTEM

Solution

Check the size and maxsize of the data files in the SYSTEM tablespace

select file_name, bytes, autoextensible, maxbytes
from dba_data_files
where tablespace_name=’SYSTEM’;

Increase the size of the data file;

alter database datafile ‘<Path to data file>’ resize <larger size>

Fixing SELinux Content in a HTML directory

Unable to access web files in a directory under the /var/www/html becuase of a 403 forbidden might be a context problem.

Problem

Getting a 403 forbidden error message when accessing a new file or directory in /var/www/html.

Check /var/log/messages to see if there are any:

Sep 13 13:11:28 fserver02 kernel: audit(1221336688.892:14): avc:  denied  { getattr } for  pid=25036 comm=”httpd” name=”phpldapadmin” dev=dm-0 ino=2441537 scontext=root:system_r:httpd_t tcontext=root:object_r:user_home_t tclass=dir

Solution

First check the context on the root directory.

cd /var/www/html
ls -Z index.html
-rw-r–r–  root     root     root:object_r:httpd_sys_content_t index.html

Look at the context of the new directory or file

ls -Z index.php
-rw-r–r–  root     root     root:object_r:user_home_t index.php

Now change the file or directory to the correct context

chcon  root:object_r:httpd_sys_content_t . -Rc

Check the context of the file:

ls -Z index.php
-rw-r–r–  root     root     root:object_r:httpd_sys_content_t index.php

 

Oracle XE Recovery

How to recover an Oracle XE Backup

First make sure the database is in archive log mode and running backups on a daily basis. See Oracle XE Backup for instructions on how to do this.

Reverting to a Previous Backup

If something has gone wrong with the database, such as deleting or corrupting a database file, and you want to restore it to the last backup run:

# su – oracle

# $ORACLE_HOME/config/scripts/recover.sh

If the database is in archivelog mode then it will restore from the last backup and a complete recovery of all the archive log files is attempted.