RSS

Category Archives: Uncategorized

Continuous Deployment via Git

Automated deployments is the holy grail of web development shops. Simple shops with only a small number of homogeneous servers have a much easier time than those of us with more complex environments. There are so many pitfalls to watch out for.  In this talk, we discuss the good, the bad and the ugly that we experienced in getting our continuous deployment process going.

Continuous integration via Jenkins had been running for months. We used it to build our svn instance and then off of our Git Master Branch for dev, QA and Staging. But getting the code deployed to 10 Data center servers and 6 Amazon servers where different servers have different configurations, operating system and yes, 32/64 bit issues, was quite a headache. Couple that with a 24×7 SAAS website with 6 Million daily visits, we had a challenge. We considered using an open source or commercial product such as Puppet, Capistrano or Maestro but they really didn’t meet the needs. They seemed either overkill or not flexible enough.

So we did the following changes to our build and deployments processes:

1. Moving from SVN cherry-picking releases to version-based promotion

In our previous workflow a release-master would take care of cherry-picking the commits associated to tickets that were successfully tested on Qa and Staging environments, and make a big merge to the next environment, all the way to production.

We moved to version-based workflow in which a new version is tagged everytime new features are merged to our master branch (using github pull requests).

Each time a version it’s approved on an environment (qa exploratory testing, product team acceptance, staging regression tests) it moves to the next one, ending up in production.

2. Decoupling build process from deployments

Updating an environment consisted of running a jenkins which will checkout a specific (environment-specific) branch from SVN, build it, and copy the output to a remote shared folder (using tools like robocopy or unison). In order to move towards continuous deployment we needed a way to deploy faster.

With the new workflow established (step 1), we started to move to each environments versions that match exactly a version deployed to dev environment, so we can now build versions once, deploy anywhere.

Decoupled build process now consists in taking a tagged version, build, and push the build output (a distributable) to a separate git repository. Git makes an excellent job in pushing only the deltas from previous version, even for binary files.
Now versions are built only once (unit tests and static code analysis tools run only once too). Using git as a store for our distributable versions allowed us to:

  • Gain ultimate traceability, now not only on source code but on deployed files
  • Mapping between deployed files and original source is a breeze, eg. this allows us to know exactly the differences between the code deployed on each environment, and other cool stuff  generating automatic changelogs based on pull request info.
  • Reduce our build time from more than 20min to < 2min

Once a version is built and pushed to git build repo, it’s ready to be deployed super-fast (a git pull of file deltas) to any environment.

Finally we added a 2nd build process using Release configuration that runs once before moving to qa environments, this one uses msbuild release optimizations and performs all client-side resource optimizations (minification, bundling, hashing, etc.)

3. Git-based deployments

Once versions are published to a git repo, they can be easily deployed using a git clone, and updated with a git pull.

To identify the versions we want to deploy to specific enviroments we use more git tags (eg. env-staging, env-production) that match specific version tags.

In order to setup a new web server we now:

  1. Clone the build repo
  2. Setup IIS website
  3. Run PostDeploy powershell script (more details below)

And each update consists in:

Manually move the env-* (eg. env-production) tag to a newer (or older) version. And on each target server:

  1. Fetch and check if the env-* tag moved, stop the process otherwise
  2. Run a predeploy powershell script, that will:
    1. Take the server out of Load Balancers (If it’s in)
    2. Git checkout the env-* tag
    3. Run postbuild powershell script, that will:
      1. Transform the config files based on the current environment name
      2. Recycle app pool if needed
      3. Run automated smoke tests
      4. Put the server back in the Load Balancers

Note: steps 2.a, 4.b and 4.d are automatically omitted if the diff to the new version doesn’t require an app restart (no .dll or .config changes)

Of course performing this process for each webserver doesn’t escalate well so in order to automate this process we considered other popular deployment tools, but none matched our needs.

We built (and open-sourced) a simplistic “deploy hub” web app that runs on each target server and updates deployed instances (running the process described above) triggered by a REST call, and can notify the result with different mechanisms.

This REST call can be done manually (curl), from a central deploy dashboard, a github post-receive webhook, or at the end of a build process.

Using git here allow us to update environments in no time (specially because as often you deploy, smaller are the deltas). This is very important in achieving zero-downtime, one of the pillars of continuous deployment.

Another consequence is rolling back is pretty straightforward (move the env-* tag and trigger a new deploy), which will happen even faster by just applying the git deltas from an already fetched version.

The overall results is TTR (Time to Release) has got extremely reduced (from hours to a few minutes), and we got rid of the stressful release day big-merges and regression nightmares.

Feedback from features comes as fast as possible, and the number of production found bugs (and the need for emergency hotfixes) reduced drastically .

Since Git does such a great job of identifying what has changed from one tag to another it was a natural way to be able to get code deployed and only have differences applied. So our Git based deployments goes like this:

1)      We set up Jenkins to pull from the Git  Master branch of our main repository.

2)      We used Powershell to script out the commands such as msbuild, git pull, git push, config transformations(cft) ,etc.

3)      With each build, we push the completely built website to a separate Repository (Builds) which only contains the build and a tag for each one.

4)      To set up the Git deploy we need to go on each server once and do a Git Clone and Pull.

5)      We also set an environment variable for each server which identifies the unique configuration for that server. Variables for each config identifies in the Environment variable include different connection strings, Some 3rd party DLL version and some domain configuration.

6)      We Built a web app that we install on each server that exposes a Rest call that will execute Powershell script that does a git Pull and runs CFT which will do config file transformations based on the environment variable to configure the specific server environment. Next the powershell script does a recycle of the app pool if DLLs are being deployed and then does a few curl requests which are some simple smoke tests to ensure the site came back up right. Once the curl requests are successful it makes a call to the load balancer to put it back in.

7)      We also built a web app that runs on a deployment server that is used to trigger the deployment. The deployment app takes a server out of the load balancer and then makes the rest call to server to trigger the deployment.

So, once QA finishes testing and blesses the release. We manually set that version with the production tag. Then we go to our deployment website, click a button and it rotates through the servers taking then out of the load balancer, calling the deployment app on each server and putting them back in. Then it emails out to tell us that it is done and if there were any issues.

Sounds clean no? Well to get there we had a bunch of pain points we had to get through.

1)      Permissions. Since we used a deployment website that calls powershell, we had to run it under an account with proper permissions.

2)      Since the deployment websites are on each web server we needed to make sure that permissions were tight. So we locked down the web site with ip and login credentials. We also only allow access to the deployment site if you are accessing it through the internal IP and via a different port other then port 80. We couldn’t use port any anyway because  the main website accepts all IP addresses into the server. So we had to go with another port so there was no collision.

3)      Because of using Git to deploy, we can not go in and manually deploy anything to the servers or the Git pull would not succeed since the server would not match the tag anymore. In the past we sometimes would take a server out of the load balancer to test a prod issue locally on the box. We needed to be aware if we do that, that we need to do a git reset to put the server back to the state that we can deploy again.

4)      Since we run a post step after deploy to set the configuration files for the environment, we needed to have git ignore them.

5)      Taking servers in and out of the load balancer that we have was not simple. There is no api for the coyote point so we had two option to take the servers out. SSh into the balancer and edit config files or do http Post commands to the load balancer.

6)      Git was always seeing DLLs as changed even when they were not. So it was always deploying the DLLs which made simple deployments of files that do not reset the app pool impossible. We had to put in a compare to verify if the dll had changed or not.

In general it took us about 2 months of deployments to get to this point after we started. We evolved to it by adding different parts each release.  It has greatly simplified our dev process and allows us to release more often with more confidence since our releases are smaller. It also enabled us to stop having to get up early to release code since we can automate the release to happen at slower times. We love Git J


 

Advertisements
 

Tags: , , ,

Bundling and async loading Javascript for fast page loading using require.js

With thanks to Benja Eidelman for help with this Post

Main Idea

Browsers delay page first render until all resources mentioned in the <head> are fully loaded. That’s why the one of the most basic tips to optimize page perceived speed is to move all scripts to the bottom of the body. The obvious next step is to remove from body all scripts that can be loaded asynchronously using one of the many script loaders out there.

We know this is only the tip of the iceberg of optimizations and fine tuning techniques that can quickly introduce lots of noise in your codebase.

We’ll share an approach to perform all common page speed optimizations while keeping your javascript and templates clean and organized.

The tools involved are require.js, grunt.js and a small library we authored to achieve clean declarative integration between templates and javascript modules.

Optimizing page speed

First stage we used require.js as an async script loader.

We got async loading (not blocking the page render or dom ready), callbacks urls, handling errors, cascading, all these with paved cross-browser inconsistencies.

Besides more control on the page load waterfall, this gives you a consistent mechanism to deal with 3rd party scripts loading errors and consistent mechanism to run JavaScript code only when a 3rd party library is loaded successfully.

Organizing the JavaScript Codebase

But require.js is much more than a script loader. require.js implements AMD in browsers. Asynchronous Module Definition is a standard way to organize JavaScript code into modules that can supports dependency resolution and prevent global object pollution.

This is a must in order to write non-trivial JavaScript apps or libraries. This gives us the freedom to structure code semantically as we do in other programming languages. Saying goodbye to the typical 1-giant-file libraries.

Having automatic dependency loading means the end of hacky async initialization mechanisms (to ensure scripts are correctly initialized when the load order is not guaranteed), like register functions, temporal polyfills or process queues.

Also by setting up require.js aliases and shims we keep our js files agnostic of infrastructure decisions (like chosen CDNs or url fallbacks).

Also this leaves us in a pretty good position to move forward when ES6 is ready, including JavaScript native module support.

Bundling

Bundling and compressing static resources, as JavaScript files is one of the basic tips we’ll get from any page speed analysis tool available out there.

The purpose is to reduce the number requests (avoiding http handshaking costs) and bandwidth (by reducing size).

Compressing is done typically by compression algorithms like gzip or image compression formats (jpeg, png), and for text files (like JavaScript or CSS) we can go further using minifiers.

Using AMD to structure code gives us the ability to create bundles automatically based on the declared dependencies, we user the r.js optimizer tool (part of the require.js project) which gives enormous flexibility in terms of configuring the creation of different bundles.

This way using AMD we can keep excellent debuggability both on development environments (where bundling is off) and on production-like environments.

When bundling is off, require.js takes care of loading the necessary modules (dependencies) asynchronously, making the file tree on browser dev tools to show the same structure and content on the source project.
When bundling is on, we use uglify2 minifier as part of r.js optimization that provides source maps. Source maps are supported in most modern browsers and allow us to debug JavaScript code on production-like environments by seeing the same file structure and code that was used to produce the bundle, breakpoints and step-by-step debugging works as expected.

The final step in our bundle process is versioning bundle files. Versioning is a critical part of deploying applications. In order to use a CDN with long cache times you need to ensure cache is purged when new versions are deployed, the safest way to do this consistently is using versioned filenames to ensure cache busting.

And using content hashing ensure cache is busted if (and only if) needed (ie. when content changes).

To keep all this process in place we use grunt.js (a node.js build tool), which provides us with a great ecosystem of 3rd party plugins for most of these tasks, and it’s pretty easy to extend.

Declarative html+js integration

Once we moved to a fully AMD-based JavaScript structure, and setup r.js automatic bundling, we realized there was an important missing piece in this architecture.

How do we make use of this modules in our html UI without introducing noise or polluting the beautiful declarative-ness of our templates.

Now that we embraced the clean code organization that AMD allowed us, we decided to keep our templates equally clean, free of embedded JavaScript code.

In order to do so, we modeled the typical cases where JavaScript modules are related to the  UI, and made them explicit in our model:

Widgets: modules that attach to a specific DOM element, modifying it’s appearance or behavior, many instances can exist on a page (a concept similar to jQuery UI Widgets)

Behaviors: modules that custom behavior that can be enabled or disabled on a page basis, they are not associated to any specific visual element. eg. Keyboard shortcuts, auto-scrolling.

View-Modules: very small modules that can wire-up elements in a specific template. (a concept similar to codebehind files in webforms)

Now these are the only entry-points in the html templates to JavaScript modules (all the other modules are dependencies of these).

And we established an html convention to declare them:

On pure html:

<h1>Using pure html</h1>
<section>
    <H2>Widgets</H2>

    <!-- simplest example, will load "user-profile-pic" AMD module, and initialize this DOM element with it -->
    <div data-widget="user-profile-thumb"></div>

    <!-- specifying parameters for this widget instance -->
    <div data-widget="user-profile-thumb" data-user-profile-thumb-parameters='{"size": "large"}'></div>

    <!-- multiple widgets per element -->
    <ul data-widget="accordion, dockable" data-dockable-parameters='{"position": "left"}'>
        <li>item 1</li>
        <li>item 2</li>
        <li>item 3</li>
    </ul>

</section>
<section>
    <h2>Behaviors and View Modules</h2>
    <!-- lists of behaviors or view modules are specified on json format in script[data-modules] tags -->
    <script data-modules type="application/json">
        {
            "view":{
                "views/home/index":""
                },
            "behavior":{
                "keyboardShortcuts":"",
                "iefixes":">=8"
                }
        }
    </script>
</section>

And using razor:

<h1>Using Razor</h1>
    <h2>Widgets</h2>
    <div data-user-profile-thumb-parameters='{"size": "small"}'></div>
    <div></div>
    <!-- Razor helper to register multiple widgets using a selector -->
    @{ Widget.RegisterByCssSelector("user-profile-thumb", "div.profile", new { size = "large" }); }
    <!-- an optional parameter (C# anonymous object) is used as default it no parameters exists on the element -->
</section>
<section>
    <h2>Behaviors and View Modules</h2>
    <!-- Activate Behaviors -->
    @{ Behavior.Activate("iefixes", ">=8"); }
    <!-- Activate multiple Behaviors -->
    @{ Behavior.Activate("keyboardShortcuts, iefixes"); }
    <!-- Load the view module associated (by path) to the current template -->
    @{ ViewModule.Register(); }
    <!-- Finally, print the script tag with all the behavior and view modules declared so far -->
    @Module.LoadAll()
</section>

Finally this data-widget attributes and script tags are parsed and modules get loaded on the client using a small JavaScript library that runs on page initialization, and supports Ajax loaded content too.

Moving towards this declarative syntax allowed used to customize our bundling process, now we analyze templates looking for AMD modules (widgets, behaviors or view-modules) and create bundles based on them (dependencies are added automatically by r.js)

Now each time a module is used on a template it’s automatically bundled without extra work.

Later, if desired, this default behavior can be overridden by moving specific modules to separate custom bundles (eg. modules that are only used on less visited areas of the site)

Conclusion:

As you can see, we found an architecture that allowed us to tune the page load towards perceived performance (the time to first render, and the time until main elements are responsive reduced significatively), while still keeping the codebase cleanly organized and optimal in terms of maintainability, by keeping configuration as lean as possible.

Here are waterfall charts of one of the hottest pages in our website:

Image

Below is the same page in dev without bundling so you can really see how require has changed the waterfall. And of course see the benefit of bundling where we save over 24 http requests.

Image

The benefit is a much faster feeling website. As you can see above on the production /live page, we are seeing start render times under a second when we still have things loaded. Start render is before the main JS file is loaded.

 

Tags: , ,

Leveraging Caching and Amazon EC2 to scale BlogTalkRadio.com from 100k to 6M visits per day

 

BTR Growth

Back in Early 2012 were looking at under 100K visitors per day and struggling with our current infrastructure. Our Databases were straining, our web servers were railing and our all important ad server was maxed out.  We knew that caching was the only answer. We embarked on a multi faceted attack to be able to support 10x growth.

We thought about three levels of caching and how to achieve it.

  1. Data Caching. Caching Data sets even if only for a minute.
  2. Dot.Net object caching.
  3. Web Page HTML caching.

Data Caching – The fastest database query is the one that doesn’t have to happen.

For Data caching, we considered the holy grail of data caching layers where every request reads and writes to the cache and hits to the DB second via the middleware.  However, we didn’t have the resources or the time to go down that path yet.  We wound up using three different data caching mechanisms which were best for each use case.

The first data caching layer that we used was within .Net itself. We used .net data caching. We decided to use this for smaller data sets that rarely change such as for configuration data and lookup tables that required expiration only when the data changed. We wanted to limit this because with .net caching, the same data is stored in memory for every process on every server. We run 3 processes in a web garden for each web server so we would be caching the same data 30+ times. For data that changed rarely and accessed often,  this made sense.

For more set based data caching we turned to  Redis as a caching data store and Postsharp as a mechanism to instantiate it. Initially we used Service Stack as the client for Redis but were not happy with the performance. When we dug into it, we noticed that service stack opened and closed a connection with every access. Wasn’t very efficient. We switched to using Booksleeve  and that worked well for a while but had trouble with deserializing nested json objects.  Finally, we turned to protobuf which not only allowed for ascii serialization, but binary which was even more performant since it made the payload to and from Redis that much smaller.

We had our home grown caching layer but soon found another method of data caching that was super simple to implement and met a need that our other mechanisms did not. We found a product called ScaleArc that sits between the application and the database as a proxy/cache layer. To use it, all you do is point your connection string to it and set it up to use your existing DBs as the origins. Then all requests are reported in nice analytics. You can then see which queries are the most expensive and then set up ScaleArc to cache the result set for that type of request. It gives great stats on DB usage and cache hit ratios. We first used it to help scale our third party ad server since we didn’t have access to the code.  Our ad server has a horrible Sql Server DB bottleneck. We were railing it constantly which caused us a loss of potential revenue. We upgraded the hardware and then soon railed that server. Since it was a third party application, we could not make any changes to the app to improve it. ScaleArc allowed us to add a data caching layer without touching the application.  This gave us a lot more room for growth and paid for itself many times over.

One of our big sources of revenue was  advertising.  We did a lot buying of keywords to drive traffic to the site.  Then in turn, we made money on the ads that were on the pages that these users visited.  These users were not the same type of engaged users that our normal registered users were, so we wanted to segment this traffic so that these visitors did not affect our normal users.  We leveraged Amazon EC2 instances to serve up that traffic.  Since our DB servers sat in our collocation space, there was a lag to access the data from EC2 to our datacenter servers. We put ScaleArc in place for the  Amazon instances for our purchased traffic and implemented heavy caching.  Since most of the traffic driven by marketing is driven initially to a set of landing pages, the db caching had a high cache hit rate and has allowed us to scale that business tremendously.

Dot.net caching – Good for some applications.

Data caching can only take you so far.  Better yet, avoid needing to require any data access at all. To this end we started caching components of a page. We mainly cached some slowly changing user controls, header, footer and a handful of fairly static pages.

When you cache user controls, we quickly learned that referencing them in subsequent page loads becomes problematic. Since it is cached, any reference to the control will return null. We could not access the control. So we had to put in some error trapping. Downside of Dot.net  output caching is that there is still work being done on the server side even though greatly reduced. It forces you to develop with specific patters to expect controls to not be referenced in your code.

Web Page HTML caching – Edge caching for the win!

Things were getting better but to get 10-20x growth, we needed to go another caching step. The best way to scale your infrastructure is to not have it hit your infrastructure at all. CDN caching was the next important step for us. We had been using CDNs for our audio files, javascript, css and images for a while, but now we set our sights on caching of full pages.

When we went to the Velocity Conference for the first time a couple of years ago, we learned about a product called aiCache.  It is a caching appliance with a lot of flexibility. It could have different caches based on cookie, browser, etc.

This worked well for us because most of our traffic was guests who were not logged in. So based on cookie we were able to direct traffic via the load balancer to aiCache if there authenticationcookie or directly to our origin servers if the user was logged in. This way we didn’t have to change the application to show personalized information.

As we continued to grow we started seeing bottlenecks with logged in traffic and the aiCache infrastructure. So we wanted to add a layer of edge caching for our high traffic pages.

We had been using limelight as our CDN for images but the performance wasn’t fantastic.  After attending Velocity in California  we learned about TCP/IP overloading and looked for a CDN vendor that did it. The overloading is much faster than standard tcp/ip because it sends packets and does not wait for an Ack before sending the next packet. We saw times of 15ms from Cotendo vs 100ms from limelight for the same cached content. We went with Cotendo which was recently bought by Akamai. (Boo)

The key with successful CDN edge caching of our pages was two fold.

1)      Proper configuration – Setting TTLs, Include Querystring, etc

2)      Client Side Customizations.

Configuration required understanding the data and how often it changed. The structure of the URLs and understanding what requests couldn’t be cached was crucial.  Many Ajax requests and some pages needed to bypass cache and go directly to the origin servers. We had to carefully set TTLs and  identify when querystring should be part of the cache key or not.

Most importantly, our pages had customizations based on who is logged in. It needed to show user driven menus that changed to show the user’s avatar, premium level, etc.   In order to properly cache pages at the edge, we had to cache a generic for a non logged in user and then apply the customizations client side.  The complexity is that you don’t want the user to see the generic page and see the customization snap in. We want the page to appear nicely for the user.

To do this we knew that it had to be done in the header with as little required code as possibly. If we waited for JQuery to load or if we made an ajax call to get information it would slow down the page or cause that snap in. So the lesser of the evils was to store the minimum needed for the header customization in a cookie and have the small js snippet that knows how to generate the header html customizations in-line on the pages.

Interesting management challenge with edge caching of most pages was the paradigm shift needed by our developers.  We had a lot of old page customization code that we had to pull out of the back end C# code and move it front end Javascript.  User-based Analytics and displaying page partials based on the user’s history and level had to all be removed from the C# and moved to Javascript.  Developers needed to switch their thinking that pages would be generic for all users.  Even months after we implemented this pattern, we were still catching mistakes during code reviews where developers were doing server side customizations rather than on the client.

With this in place we were free to launch the CDN edge caching of HTML and our logged in users saw what they expect to see. All was good.

Because of the increased cookie payload, we made sure to move most of our static file http requests to cookieless domains (btrcdn.com) so it wouldn’t add a lot more overhead to the http requests.

Adding all of this caching enabled us to grow our traffic over 10x and still have plenty of room to grow another 10x.

 

Blackin Family Trip to Israel

The entire Blackin clan embarked on an adventure on June 23rd, 2012.  On the trip were my Mother, Father,  wife, three kids (13,12,9), my brother and his four kids (10,9,6,4). Table for 12 please was our battle cry three meals a day.

When we travel with my parents, we do it right. We stay at the nicest places, eat the best restaurants, and have a private, air conditioned bus for traveling around. This is in stark contrast to the first time I went to Israel.

First time I was 17 years old and went with the a terrific youth group called USY.  We had 80 high school kids, and we stayed in youth hostels and kibbutzim. That was more of a third class trip as opposed to the first class trip we took with my parents. The itinerary with USY was much larger and in depth since it was for 6 weeks as opposed to 10 days with my family.

June 23, 2012

We arrived at Newark Airport for our 3:55pm flight and took off “on time” at 4:30 ish. We started on Jewish time right off the bat. P1010739

Trip was happily uneventful. Everyone was very happy with the in-flight entertainment. Between the 166 movies and dozens of games, there was enough to keep the most picky 13 year old and care free 4 year old satisfied. Unfortunately, due to the high level of entertainment, very little sleep was had on the flight.

We flew over the alps and saw some spectacular terrain.

P1010748 ______________________________________________________________________________________

June 24,2012

We landed around 10:30am at Ben-Gurion airport in Israel. After gathering our luggage we met our tour representative and boarded our bus. P1010756

We arrived at the Crown Plaza Hotel in Tel Aviv, right on the beach. The kids went wild “Lets go to the beach!”. So we got our luggage up to our rooms, changed into our bathing suits and headed out onto the beach. The Mediterranean is amazing. Such a beautiful blue color. P1010757 As if designed for our family, the depth of the water increased very slowly. Therefore even the youngest of our crew could walk out 20 yards into Mediterranean comfortably.

P1010759Kids had a great time on the beach. Considering most of them, and us adults, were awake for over 30 hours, we did quite well,P1010760

We got the kids off the beach around 3:30 so everyone could shower and we could get an early dinner. We had a delicious dinner at a place called Jessica’s. Only the two youngest fell asleep at dinner, My brother Mike’s sandal broke on his walk from the hotel to the restaurant. I called it a flat tire.  After dinner we tried to find a store to get him a new pair of sandals. We asked a couple of Tel Aviv residents, with Maya’s help where we could buy shoes.  However, even after trying to follow the directions, we were unable to find the store and we got back to the hotel with a shoeless Mike.

We went up to our rooms and very quickly fell asleep by 8:30.

__________________________________________________________________________________________

June 25, 2012

We got up around 7am to get ready for our first full day in Israel. Plan was to tour around the Tel Aviv area. We  hit the breakfast buffet and was incredibly impressed with it. The buffet definitely was up to the U.S. standards of breakfast buffets. P1010767 But it had an amazing view of the Mediterranean.P1010768 Our first stop was….a mall to get Mike some sandals! He was admonished at breakfast for going into the restaurant without shoes.

Once shoed up we headed on a driving tour of Tel Aviv. One stop was at the memorial for Yitzchak RabinP1010769 P1010771

Yitzchak Rabin was the prime minister of Israel in the mid 90’s. He was assassinated by an Israeli because he was working towards peace with the Palestinians and was discussing giving them land. Some Israelis were not happy with this and plotted to kill their own prime minister.

Lunch was just what we wanted. We went back to where we bought Mike’s shoes because there was a Shwarma and Falafel store there.P1010786 Leah had been asking “What is Shwarma?”. Well that answer was answered with a huge wrap. Everyone enjoyed!

 P1010787 P1010785

Since Israel is mostly a desert country where is hardly ever rains between April and November, beautiful flora is not naturally occuring in great quantities. To solve this problem, Israel pioneered a drip watering system all of the country to water plants efficiently so they do not waste precious water.P1010772 The most interesting stop of the day was at the Ayalon Institute. Back when Israel was still under British rule between 1945( when WWII ended) and 1948(when Israel became an official country), a most clandestine operation took place on this kibbutz. 

P1010776

45 of the 75 members of the Kibbutz were involved in this most illegal activity. Had they been discovered, by the British soldiers, all them would have been killed. They were making bullets. The Haganah was Israeli clandestine organization that knew that after WWII, that the British would be leaving and a war for their existence was going to happen with their neighbors. They charged these 17 and 18 year old settlers of the Kibbutz with this dangerous job. To keep it secret from the world, they build their factory 50 feet underground. It could only be accessed by moving an industrial washing machine on one end and a ten ton bakery oven on the other end. Over the three years that the factory was active they made over 5 million bullets. This contributed greatly to the success of Israel’s independence.

P1010778 P1010777

After the bullet factory, we headed down to Tel Aviv University where the diaspora museum resides.

The Diaspora Museum is very interesting. The basic premise is that Jews have been spread all over the world because of all of the times that our people were kicked out of counties or fled to avoid various persecution.The big lesson is that even though we may be geographically diverse, we are all connected by traditions. A Jew in Ethiopia follows the same general traditions as a Jew in USA or one in Argentina. All will get married under a chuppa and have a Passover Seder no matter where you live.

P1010794 P1010793

After the Diaspora museum we drove to Jaffa and walked around the old city. It is quite beautiful and hilly. The view back to Tel Aviv is spectacular.P1010820

We got some ice cream P1010818and the kids pet every cat they could find (there were a lot of them) There were also some interesting art,

P1010805 P1010811  P1010802

After Jaffa we went to dinner at the waterfront. We ate a nice restaurant with the same name as our friend.

P1010821

Then we walked back to the hotel along the promenade.

P1010826  Our friend Nimrod, stopped by for a drink or two with us. He is an Israeli who lived in the US back in the 80s. Nimrod  went to high school with Mike and college at NJIT before moving back to Israel.

P1010827

________________________________________________________________________________________

June 26, 2012

We got up early and packed. Had another breakfast that can’t be beat and loaded up the van northward.

Our first stop was Caesarea along the coast, north of Tel Aviv. Caesarea was originally built by King Herod around 22 AD as a deep water port. Over the centuries it was destroyed and reused by different factions based on who controlled the area.Roman, Muslim, British, etc. Since it was a center for trade, people coming needed entertainment. There was a hippodrome for chariot races and gladiator fights.and an amphitheater that they have restored and are using again,

P1010831 P1010839

Next we headed to Kibbutz Ein Shemer to learn about kibbutz life and lunch. The kibbutz started around 1927 as true communal life. Children slept in the Children’s house and did not sleep with their parents. All money made went to the Kibbutz and the Kibbutz would give  money per need. Everyone was treated equally. Within 2 generations the kibbutz changed to be more like a moshav. People kept all of the salary they made ad paid a tax to the Kibbutz for common areas. This kibbutz  was primarily a steer and milk business. The kids enjoyed petting the calves and watching the machines milk the cows. Boy did the cows stink! We also had lunch in the communal dining hall.

P1010856P1010859 

From the kibbutz we headed to meet our cousins at Shavei Zion. We were very excited to see and meet our cousins. The 4 who I had met before are my third cousins David, Dori,, Estee and Shai.  They have 7 kids among them.

P1010877

David and  Yona have two girls, Gal and Eden.

 P1010878 

Dori and Marev have three kids. Amit (B), Shira (G) and Yishai.(B) Maya and Shira were immediately attached at the hip yapping away in Hebrew.

 P1010889P1010893

Estee and Rami have two kids. Alon (M) and Noam (G), They are so adorable.

P1010894 P1010868 P1010869

And Shai who is not married as of this publication.

We had an amazing meal with mini burgers, chicken kebobs, chicken wings, several yummy salads, homemade hummus and wine.

Dori took us all to the beach. Kids played with boogie boards, dug in the sand and got to know each other better.

P1010880 P1010890 P1010885

Unfortunately David’s family had to leave to go to a wedding as did Shai so they were unable to stay for dinner.

We took a picture of all of the kids that were still there at the end. Missing is Gal and Eden. We expect some of them to become e-mail pals and keep in touch.

P1010895 

After dinner we headed to Haifa to the Dan Carmel hotel. Wow, what a gorgeous hotel. We had an unbelievable view of Haifa from the top of the mountain. 

___________________________________________________________________________________

June 27, 2012

Happy birthday to my nephew Daniel.

We got up and packed since we were only staying one night. We went down to breakfast and again were pleasantly surprised at how good it was. This one had an omelet station where they made omelets to order. Yum.

We headed out for a driving tour of Haifa and stopped at the Baha’i Shrine gardens. Our tour guide explained that the gardens were built by a Muslim guy who claimed to be the new prophet. He started a new religion and this was built to honor him. It is quite beautiful. When I was here the first time, we were able to go into the gardens. Now you can only view it from the top or bottom.

P1010902 P1010907

After the Baha’i Gardens we drove to Akko (also called Acre). This another port city like Caesarea where control of it changed over the centuries. Originally built as an international deep water port in second century BC, it was taken over by the crusaders in 13th century when they built several fortresses. The ottomans in 18th century neglected it and filled in all of the old fortresses, tunnels and dungeons with rubble, P1010929

The British used it as a a jail during their control of the area. In the 1960s the excavations began in the area and all of the old layers of buildings, tunnels and fortresses were unearthed.

P1010931 P1010985 P1010988 P1010911

It was here that we did some haggling and bought Serena an interesting necklace. It is a Star of David with two side pieces that when brought together makes a Hamsa.

P1020150P1020149 

Since today was Daniel’s birthday, at the shop that I bought Serena the necklace, an artist made a nice surprise for Daniel. Everyone was mesmerized watching him make it.P1010953 P1010955

From here we went to lunch. We filled our bellies with delicious fresh lemonade, shwarma and falafel,  At lunch we met a family from Boston there on a Bat Mitzvah tour. The mother graduated Tufts 2 years after I did. We coincidently saw them again at Rosh Hanikra.while waiting in line for the cable car to take us down to the grottos. We may bump into them again in Tiberias.

After lunch we headed north to Rosh Hanikra. Our bus driver got pulled over by a an Israeli policeman for almost forcing the cop off of the road. He did not get a ticket as he explained that he was in the Bus’ blind spot and he did not see him till it was too late. Since he had a clean record he did not get a ticket.

Rosh Hanikra is on the Lebanese border. The only evidence of this is a row of buoys and a cable in the water as well as a big barbed wire gate. All was quiet on the northern front. What wasn’t quiet were the three bus loads of South American teens on line to get on the cable car to descend to the grottos.

P1010994 P1020013

P1010992 P1010993

 

The grottos were worth the wait. Absolutely beautiful. Not just the grottos themselves but the surrounding Mediterranean.

P1010995 P1010999

P1020003 P1020009

After Rosh Hanikra we boarded the bus and headed for Tiberias, We arrived at 5:30pm at the Rimonim Galei Kinnereth right on the Sea of Galilee. The contrast of the blue water and the desert mountains is quite impressive.

P1020020 P1020019

At dinner, since it was Daniel’s birthday, we sung Happy Birthday in Hebrew and English and they gave him a sparkler in the dessert which he tried to blow out.

P1020025 P1020027

After dinner we went outside and watched a water/light show over the water. Then we went to bed. P1020033 P1020035

__________________________________________________________________________________________

June 28, 2012

We woke up, had another great breakfast and headed out to the bus. We had a full day ahead of us.

We took a boat ride on the Sea of Galilee / Kinneret. We had the boat all to ourselves. We viewed the sights and danced to Jewish music as we sailed on the Kinneret. Fun was had by all.

 P1020039 P1020066

From here we went to plan a tree. A Mitzvah that everyone should do to help add forestation to the desert country. There are drip pipes all along the rows where we planted so they could be well watered and grow to be mighty trees.

P1020075 P1020076

Next we headed to a shooting range to shoot GUNS! Everyone was so excited to shoot those hated targets.

All of the kids, Serena and my mother shot .22 rifles. Dad and I shot a Glock and Mike  shot a 9mm. Those targets didnt stand a chance.  The Glock surprising had almost no recoil. It felt so smooth. Leah, after her turn declared that she wanted to buy a gun and go to shooting ranges at home. Great fun was had by all.

P1020110 P1020105

After the adrenaline wore off we headed to Safed, an artist colony, Here we saw spectacular views, beautiful art and mass produced art.

 P1020126P1020114

We wanted to buy something for the house, but, I wanted something locally made. We found a painter who had lovely water color and oil originals. The artist was on site, selling his own wares. We saw a watercolor of a representative street in Safed but there were no people in it. I mentioned that I would buy it if he had people. He said he could add the people in for us. So while we watched, he added a father and child to the scene.  P1020130

We also visited an ancient Sephardic synagogue, lots of stores and a talis making factory where the tallit were hand made on a loom

P1020120 P1020123

After Safed we headed back to the hotel in Tiberias. We had a buffet dinner with tasty food, sat outside for a while and went to bed.

_____________________________________________________________________________________________

June 29, 2012

Last day in Tiberias! We packed up and after breakfast we headed out on the bus. First stop was Kfar Kedem. They make the past come alive. We all dressed up in period garb from 1800 years ago. We talked about the fact that the Mishna was written at that time by one of the residents of the area. The Mishna is the traditional interpreted laws of from the torah such as Bar Mitzvah at 13 and weddings under a Chuppah.P1020164

We milked a goat, made goat cheese, cooked pita and ate some yummy hummus.

P1020173 P1020182

P1020192 P1020195

After the snack we loaded the younger folks onto donkeys and went on a donkey ride. After the donkey ride we washed up and hit the road.

 P1020198 P1020201

Next we went to Beit Shean. This is an ancient city from the time of King Saul. We took a tractor ride around the old Roman ruins. We looked at an old map of the city and saw how much of the city still needs to be excavated.

P1020206 P1020207

We walked over to a mall next to the ruins and ate some lunch. Some kids had Kosher McDonalds, some of us had pizza and some had Falafel.

P1020220 P1020219

We stopped the lovely oasis called Gan Hashlosha (Sachne) for a swim. It is a a series of natural pools connected by man made waterfalls and broken down slides. In theory this sounded great. Problem was, there were a ton of people and the water was a bit polluted. There was a ton of garbage and Rachel cut her foot on broken glass. We were supposed to stay till 4pm but left early.

P1020223 P1020222

After we left we headed to Jerusalem. We stopped quickly a road side store and to let the kids ride a Camel. They had two camels and all of the kids got to ride and scream. They screamed because when the camel sat down, it drops it front legs first so everyone feels like they are going to fall forward.

photo photo (2) photo (1)

We passed by many Bedouin villages. They live in make shift tents. Some made of scrap metal and parts of things they they found laying around. We saw lots of sheep and goats on the desert mountains of the west bank.

P1020236

The kids cheered when we come up to Jerusalem and saw the old city and Dome of the Rock.

We arrived at the Inbal Hotel in Jerusalem and had a nice dairy dinner in the plush lobby.

P1020230 P1020232

_____________________________________________________________________________________________

June 30, 2012

.Had a nice cold breakfast. Since it was shabbos there was nothing hot. Were plenty of choices.

Today is a big day. Everyone was excited. We headed towards the Dead Sea. Along the way we passed the Qumran caves where the Dead Se Scrolls were found.

First stop Ein Gedi! It is a very hot day today (108) so we thought we would start it off with a swim in a waterfall along the Jordan River. The juxtaposition of the lush flora along the waterfalls with the desert limestone and Dead Sea in the distance was impressive. We enjoyed playing in the waterfalls before heading off to Masada.

P1020272 P1020256

P1020273 P1020269

Next Stop Masada!  It was very hot so we eschewed the snake path and took the cable car up to the top. Here we walked around a little and looked at the map of the top that they kindly placed under a shade tent. Then we crossed over to the other side and looked at the remains of the ramp that the Romans had built to break into Masada.

I have been on top of Masada twice before and know the famous story well. However, learning the history of who has used this flat mountaintop is interesting.

  • Jews first fortified Masada during the years after the Maccabees drove the Greeks out of Israel and became masters in their own land.(100 BCE)
  • King Herod, who ruled Judea on behalf of the Roman Empire at the end of the first century BCE, was the next to build up the mountain.
  • Following Herod’s death, a Roman garrison remained to guard Masada. At the beginning of the Great Revolt (67-73 CE), in which the Jews of Israel rose up against the Romans. A band of daring rebels overcame the mountain’s guards and captured Masada. These are the famous inhabitants that the stories are about.
  • During the fifth and sixth centuries, monks lived in caves and cells on Masada
  • The site of Masada was identified in 1842 and extensively excavated between 1963 and 1965
  • In 20th century Masada became a symbol of courage for the emerging modern Jewish state. It was a pilgrimage site for youth movements and Haganah members. In 1949, at the end of War of Independence, the Israel flag was hoisted on Masada’s summit.

P1020277 P1020289

P1020290 P1020283

Next Stop Dead Sea! We were all looking forward to this. Unfortunately some members of our family had cuts and scrapes that kept them from truly partaking of the only unsinkable body of water in the world. Those of us who went in enjoyed a good float. We even floated vertically after floating out to the deeper water.

The Dead Sea is 1200 feet deep at it’s deepest part. The bottom, which we quickly discovered, has large salt crystals everywhere that felt like jagged rocks when stepped on. So we walked out a few feet before the rocks began and floated out. All we needed was a foot of water before floating on our backs out to sea.

P1020299 P1020300

Kids had fun with it as did the adults.  The Dead Sea water was very warm. Had to be at least 102F. The Dead Sea water minerals seemed to really retain the heat. Sometimes we would hit a cool spot while floating and my brother felt that it may have been where someone peed in the Dead Sea.

After floating for a time we headed over to the pool and splashed about and played.

P1020309 P1020306

After Splish splashing for a while we header back to the the hotel for dinner and bed.

I had heard about the Israeli Microbreweries popping up around the country and luckily found one at the bar in the hotel. It was not on the menu. I happened to see it in the refrigerator behind the bar. It was quite good! The beer came from Jems Beer Factory in Petach Tikva.

_________________________________________________________________________________________

July 1, 2012

We had our usual nice breakfast but this time with hot food. I had an omelet. Then we met the bus and heard about the fact that former Prime Minister and giant in helping to found the State of Israel, Yitzhak Shamir died at the age of 96 and would receive a state funeral tomorrow.  Our tour guide indicated  that we had to change our plans for today since much of the city will be closed for the funeral.

We started out the day in the Southern Wall Excavations at Jerusalem Archeological Park. Here they are excavating the Southern part of the temple mount wall and the rest of the the western wall that is south of the Kotel. This western wall is the same wall that we celebrate as the Kotel but is not free to access. Also, the Kotel is the closest point that is easily accessible to where the 2nd temple once stood. We also learned of Robinson’s arch which was a gateway into the western side of the It was built to link the Tyropoeon Valley street, a major traffic artery in the Second Temple Period, with the Royal Stoa at the southern end of the Temple Mount platform

P1020332 a drawing showing a wide staircase leading up from a crowded street then turning right with steps ascending over a series of arches before turning right once again and going over a final large arch spanning the street below and passing under the columned gateway high above the streets below

The part of the western wall that we saw in the excavations is an egalitarian section where men and women can pray together. The holier part, the Kotel has women and men separate.

 P1020343 P1020338

Just as we saw that the Kotel is just a section of the western wall and that there is more of the retaining wall all the way to the southern end, now archeologist have found that the retaining wall, all the way to the north end, is in tact. Since people live right up to the temple mount where the Dome of the Rock exists, Archeologists have been unable to see what lies below. Until now. Archeologists have tunneled under the city all along the western wall creating the Western Wall Tunnel. We went under the city to walk along the rest of the Western Wall. At one point there were people praying  in the tunnel. This is the actual closest spot on the western wall to where the 2nd temple was. We saw cisterns, mikvahs, and old city streets along the western wall tunnel.

Another interesting thing is that the second temple was only in use for 70 years before the Romans came in and toppled the city. They knocked down the walls in the temple mount but did not destroy the retaining walls. The retaining walls, which were an incredible feat of engineering were designed to level the temple mount despite the northern and southern ends being of considerable different altitudes. King Herod was quite the architect. In fact, the style of having a smooth border and an rougher outset middle is called Herodian style.

P1020355 P1020360

Next we went to the Israel Museum. Here we had a quick lunch and headed in to view the scale model of Jerusalem in the time of the Second Temple. This was quite interesting considering we had just viewed the ruins earlier in the morning. So seeing the model with Robinson’s arch and Wilson’s Arch after seeing the ruins was quite timely.

P1020374 P1020368

After discussing the model and more about the second temple time, we turned our attention to the Dead Sea scrolls at the museum, also called the Shrine of the Book. My kids had visited the Dead Sea Scrolls when they were in NY but Serena and I had not.  Essentially, scrolls were found in clay pots in caves near the dead sea. The scrolls were written in normal Hebrew. Most of the scrolls contained laws for an ultra-sect of Judaism called Qumran. They fled Jerusalem during the 2nd temple time and hid out in caves. They wanted a much more orthodox, and much more strict way of life, so they moved to do their own thing, They hoped that one day, they would be able  to move back to Jerusalem and spread their extreme teachings. They made the Hasidic seem too lax about things.

Anyway, they were all killed when the Romans, enroute to Masada, found them and wiped them all out. Only the clay pots with their teachings remained.

We drove passed the Knesset, the Israeli Parliament building and all of the other ministry buildings.

P1020376

We arrived at Yad Vashem, the Holocaust Museum. You need to be over 10 to enter since there are a lot of vivid pictures and descriptions of what went on. After much discussion, only My parents, Serena, Rachel and I went. Everyone else headed to the hotel pool.

It was very interesting. I had been to Yad Vashem twice before and to the Holocaust Museum in Washington DC before. This one spent more time on the ramp up before Hitler implemented his “Final Solution”. Showed how he slowly and methodically, since the 30s started taking away liberties from the Jews until he took their lives.

P1020377 P1020378

Overall I was disappointed with how they had redone it. There wasn’t the same emotional connection that we had felt before in other museums and in Yad Vashem of old.

We headed back to the hotel and rested for an hour before heading out to dinner at a meat restaurant across from the hotel called Olive and Fish.  After eating dairy for 3 days, everyone wanted meat. My brother only wanted to find a restaurant where he could get bacon. I told him that would be near impossible. Dinner was yummy, I had Mango Chicken and a micro brew from Shapiro Brewing company right in Jerusalem, Again, we had to ask if they had any of these micro brews because it was not listed on the menu.  The micro brews are WAY better than the mass produced beers.

P1020379

After dinner we went to the Sound and Light Show at the Tower of David Museum. The Night Spectacular moves us through the sands of time chronicling the history of the city of Jerusalem from Creation to modern day as told by the pages of countless books. It was quite cool how they used the walls of the citadel as the canvas to project the scenes. Below is the same citadel, the left is normal and the right is the projection of the garden of Eden time.

P1020392P1020394 

After the show we hopped into cabs and headed back to the hotel and to bed.

__________________________________________________________________________________________

July 2, 2012

After breakfast we headed out to the bus. First stop was the Hadassah hospital and the Chagall windows. Unfortunately, no pictures were allowed. I did take some from the outside but it doesn’t show much other than how they are put together.

P1020397 P1020402

The 12 windows represent the 12 tribes of Israel. Cool thing is Rachel’s parsha for her Bat Mitzvah was Jacob telling his sons what he thinks of them. The windows depict the sons and things about them. It was nice to relate it together.

We hit the mall attached to the hospital for some needed items and some Hebrew books for Maya. Cool seeing popular books in Hebrew. One of the books that she bought was Captain Underpants in Hebrew.P1020403 P1020404

We stopped at a park for about 15 minutes where a famous Monster Slide resides. Maya had heard about this park from her teacher and the kids had fun with the cool triple slide.

P1020410 P1020407

After the park we headed over to the Kinnesset and the big Menorah. The Menorah is the national symbol for Israel. The Menorah has many famous biblical scenes on it. When we were there it was Yitzhak Shamir  funeral. The flags were all at half staff and the security was extra tight.

P1020421 P1020420

FYI – A menorah has 7 branches. A Chanukkia has the 9 branches (8 plus shemash).

After the Kinnesset, we headed over to the Old City. We walked through much of the Shuk and had lunch at a place the our tour guide Yahaloma (Diamond) brought us to. It had fantastic Shwarma. The lamb Shwarma was fantastic. Mike bought pizza from Jerusalem Pizza for the kids who didn’t like the Middle Eastern food.

P1020424 P1020423

After lunch we went shopping in the Shuk. Kids got to witness the bargaining cat and mouse game that both sides seem to enjoy. We bought a couple of beautiful dresses for Rachel and Leah. We bought the pair for 200 Shekels, 100 each.  The price he originally quoted was 320 shekels and he said it was his best price and he is making no money on it. Gotta love the lies and deceit. Since I didn’t have 200 shekels in cash the next game how to pay began. He took me to a money change place which had a crappy rate.  Learning from what I saw happen to my brother, I made them charge my card in Shekels rather than dollars so the bank would convert rather than the rip off booth. After some back and forth they capitulated and the deal was struck,

We also bought a Kafir for my brother in law. That was another fun one. Started at 60 shekels and finally purchased at 30 after I had walked out of the store.

P1020425P1020428

P1020453

After we made it out of the Shuk with our lives and dignity we walked through the Jewish Quarter and talked about the recently rebuilt Grand Synagogue and why there is a mosque in the Jewish quarter. We exited at the Zion gate. The gate area is riddled with bullet holes from the War for Independence in 1948 when the Israelis battled the Jordanians here for control. Israelis were outside the gate battling to get in. So the realization here is that the bullets manufactured in Ayalon Institude, in the bullet factory before the war could have made these holes.

P1020435P1020438

Also, if you ever go to Israel, be sure to buy water bottles at stores and buy it in packages of 6 big bottles. This costs around 16 shekels for the 6 bottles where buying one bottle half that size in tourist places and restaurants costs 8-10 shekels for one bottle half of the size of the one in the six pack. 

After the Jewish Quarter we hit the road again over to the Time Elevator at Beit Agron. The Time Elevator is a great experience. It is a multi-sensory presentation that takes you on an exciting journey through 3,000 years of Jerusalem’s history. This tied in very nicely with the Light show we saw the night before. They both took you though the history of Jerusalem in different ways.  We all enjoyed that Topol played the main character in the show since we all love Fiddler on the Roof and Flash Gordon.

P1020443 P1020442

After the Time Elevator we headed back to the hotel and went for a swim.

P1020449P1020451 

Then dinner downstairs. Now that it is not shabbos, we were able to order the hot food. So during our meal we watched a dozen orthodox couples come in and and talk. Looks like date night at the Inbal hotel. I saw one young couple come in with an older man. I assume that it was the father of the bride and the young man was asking him for his daughter’s hand. We enjoyed making up  little stories about what they were talking about and how the date was going.

After dinner we went up to bed. We are all very tired from touring all day, every day. I am going to need a vacation from this vacation.

________________________________________________________________________________________

July 3, 2012 – Last full day in Israel 😦

After breakfast we headed to an active archeological dig at the  Tel Maresha Dig at Beit Guvrin National Park.

In the late Persian period a Sidonian community settled in Maresha, and the city is mentioned in the Zenon Papyri (259 BCE). During the Hasmonean wars Maresha was a base for attacks against Judea and eventually suffered retaliation from the Maccabees.We excavated caves that had been filled in by  people who lived there when they left the area. They had filled in the caves with everything that they didnt take with them,

Everyone digging found at least some pottery pieces. My brother found part of a tool. Leah found part of a jug that contained wine. You can see the wine sediment on the piece. Everyone had fun digging in the dirt and uncovering 2000 year old treasures. We learned that these caves had been used for many purposes. Cistern, grain store, olive store, raising pidgins, and many many  more.

P1020471P1020477 P1020482  P1020480

After digging and separating dirt into buckets and “finds” into other buckets we carried the dirt buckets up out of the cave and sifted through the dirt. We could more pieces of pottery, bones and shells that were missed in the initial inspection.

 P1020485P1020484

We then went spelunking through candle lit caves that haven’t been analyzed yet. We saw pidgin holes all over these caves. We had to bend, duck, crawl and drop to get through the maze of tunnels. After 20 minutes going through the caves, we wound up surfacing from a hole only 30 feet away from the one we started in.  I cant believe the small openings that I had squeeze myself into.

P1020487 P1020488

After washing some of the dust off of us we headed to Latrun for the Armed Forces Tank Museum.

This museum has dozens of retired tanks ranging from the 1930s up to today. Before being a museum, this site was the border with Jordan and a key battle in the war of Independence in 1948. Now it contains a memorial to all tank soldiers who perished in service to Israel and the old tank museum.

Kids had a lot of fun playing on the tanks. They were able to climb all over them. But they were not able to go into the tanks. Much to their chagrin.

P1020499 P1020503

P1020505 P1020502

.At the end of the tour we said goodbye to our tour guide Yaholoma (Diamond) as she went off to get ready for her next group. We boarded the bus and our driver Mickey took us back to the hotel.

P1020508P1020510  

After returning to the hotel I went out and met my friend Josh whom I worked with 5 years ago at IDT. Was very nice to see him again. He bought me a nice coffee 🙂

P1020516

Dinner was in the hotel and then off to bed. We had to get some sleep as we needed to get the bus at 7am to head to the airport to fly home.

P1020518 Fly home, not horsey home!

___________________________________________________________________________________________

July 4, 2012 – Fly home

 

____________________________________________________________________________________________