Wednesday 18 July 2018

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

The links to the commits are probably dead because I rebased the branch. Check the branch for all the commits

August 14 - 17:

Commits:

Summary:

  • Started converting the transmit side of netfront.
  • Almost complete. I can boot, and use ping without a problem.
  • But when I tried to scp a large file from one system to the Xen DomU, SSH fails to connect.
  • Also, if I try to send a file, it panics. Sending works fine without the transmit patch, just the receive patch, so it is a bug I introduced with the transmit update.
  • I will try to fix it in the upcoming weekend.
  • This is the last part. Once I have fixed the bugs, I’ll ask Roger to review. After this, we can start the process of getting it merged upstream.

August 10-13:

Commits:

Summary:

  • Minor fixes.
  • Almost finished updating netfront’s receive side of things to use the new busdma implementation.
  • It works, but only partly. The kernel boots and connects to xn0 without any problems. Also pinging also works.
  • But when I tried to scp a large file over, the kernel panics because of a failed assertion. This is probably because I have not yet figured out how I can update xn_rebuild_rx_bufs().
  • Fixed the failed assertion. It was because of incorrectly indexing the map we needed to look up.
  • Now I can download large files without a problem. Tested using scp and pkg install.
  • More small fixes and readability improvements.

August 6 - 9:

Commits:

Summary:

  • Partially implemented multi-load support, when I realized a flaw in my plan.
  • I was planning that we create one map for each rx or tx queue, and call load for every grant reference we need. Whenever we need a grant ref, we load exactly one page, so one reference is used. The call to xen_dmamap_get_grefs() would return the most recently allocated grant reference. This way we can get the reference.
  • The problem comes with unloads. We can’t unload parts of the map individually. An unload just unloads everything. So, we can’t individually free grant references. This essentially makes it impossible to maintain a pool of grant references, like netfront does.
  • Saved the partially completed code in a different branch, in case I need it later. Check it here.
  • So I have now decided it would be better to just create a map for each grant reference, like OpenBSD does. I will need to confirm with Roger that he is on board with this idea. I have already started updating netfront in the meantime.
  • Added the ability to reserve grant references on map creation.

August 4 - 5:

Commits:

Summary:

  • Spent some time cleaning up the code. Reading the style(9) guidelines, I realize I have been doing some things wrong. It says tab width is 8, and I have been developing with a tab width of 4. Because of this, some lines cross 80 columns. Fixed them.
  • Fixed some more minor style problems.
  • Documented the reasons behind using temp_segs in a code comment. Roger asked me about why I was doing this, because even he was confused, so I guess for anyone unfamiliar it would make absolutely no sense. Hopefully it would be easier to understand now.
  • Fixed a potential deadlock between xbd_io_lock and gnttab_list_lock. I am still in discussion with Roger about this, but his suggestion to use a taskqueue(9) wouldn’t work because the deadlock would happen before we ever get the chance to get to the callback. So the fix has to be in lockfunc supplied to the tag creation.
  • Started adding support for loading an already-loaded map.
  • Down with fever on Sunday. Wanted to complete support for loading already-loaded maps, but couldn’t.

August 2 - 3:

Commits:

Summary:

  • Two rather busy days of college. Couldn’t do as much as work as I would have liked. I’ll try to compensate it this weekend.
  • Discussed the issue of locking order with Roger. The conversation is still going, but I have a good idea on how to fix it. Let’s see if Roger agrees.
  • Wrote comments that explain why we need to copy the segs array into temp_segs. Roger asked me about it in a review, and it prompted a long discussion about it. It is rather confusing to anyone not intimate with the implementation, so a detailed comment would hopefully shed some light on why we do it.

July 31 - August 1:

Commits:

Summary:

  • Finally able to completely test the busdma implementation in the shortage of grant references. After some fixes, it works.
  • But there is a problem: if one map is waiting for grant references, and another map calls unload, making ample grant references available, the callback will be called with xbd_io_lock held. But xen_gnttab_free_callback() tries to acquire the lock before calling the client callback. This would result in a recursion on the non-recursive lock causing a panic.
  • But the lock should be held when xbd_queue_cb() (the callback specified to load by blkfront) is called.
  • Make xbd_io_lock recursive. It works, but I get a lock order reversal warning. From what I understand about lock order reversal warnings, they mean that if I was unlucky, I could have caused a deadlock. Looking in the locking order, sure enough, it is possible to deadlock in the following situation: One thread holds xbd_io_lock and it tries to allocate grant refs, so it tries to acquire gnttab_list_lock. But gnttab_list_lock is held by another thread that is currently freeing some grant references. If in this situation, the free callback is received, the thread holding gnttab_list_lock will try to acquire xbd_io_lock, causing a deadlock.
  • Need to ask Roger how I can fix it.

July 29-30:

Summary:

  • More testing.
  • Finally able to produce a shortage of grant refs.
  • The kernel panics when the grant table callback is received because of a recursion on the non-recursive lock gnttab_list_lock. More details here.
  • Fixed it by adding a check in get_free_entries() about whether the thread is holding the lock or not. If it is, don’t try to acquire the lock again.
  • Roger thinks it is better to make it a recursive lock. Submitted a patch. Accepted and committed by Roger.
  • More testing reveals that I probably got lucky when I was able to receive the grant table callback, because right now, the kernel deadlocks (I think) whenever I try to produce a shortage of grant references. It is either a deadlock or the busdma implementation is not releasing the grant references properly. Will need to look into the blkfront code to see where things are going wrong.
  • Testing for low grant refs has low return on effort I believe. Before the patch I submitted, it was IMPOSSIBLE to use the callback without causing a panic, and it never got fixed until today. So apparently shortage of grant references does not happen very often.

July 26 - 28:

Commits:

Summary:

  • It is really tough to balance GSoC and college. But I just have to suck it up. Working 75+ hours a week is really tiring.
  • Trying to produce a shortage of grant references has led me on a long chase. Turns out, there are more bugs in the Xen code (the code that is already there, not the one I’m writing) than I anticipated.
  • If USB is allowed to have root mount hold, and there is a situation with scarcity of grant references, there is a kernel panic originating from an incorrect list removal from usb_bus_explore()'s call to root_mount_rel().
  • I have no idea how to fix it. It is not directly caused by the grant table subsystem, so I’m lost.
  • Circumvented it by setting USB_HAVE_ROOT_MOUNT_HOLD to 0 in dev/usb/usb_freebsd.h.
  • It is a lot of trial and error involved here. It turns out, if I have a maximum of 8 grant table pages, I can boot a DomU and produce a shortage of grant refs.
  • Luckily, I just discovered a double free which happens when a disk is removed on a running system. Check the commit here for more details.
  • Running multiple block devices at the same time is proving to be a little complicated. I want to try with one disk first. Then I’ll go to multiple disks.
  • Ran make -j4 kernel to see if the build completes. It goes on for a while but then freezes because of a shortage of grant refs.
  • If I do ^C, it gets me back to the shell, but any disk operation permanently freezes the system. Maybe the grant references are not being freed properly?
  • I’ll admit, I am getting a little bored of this shortage of grant refs thing, but I have to test it. But, as a change, I started looking at the netfront code.
  • It is rather straightforward, but the current implementation can’t be used with netfront. That is because right now, both allocation and mapping of grant refs happens at the same time. This won’t work with netfront. This is because netfront maintains a pool of grant references upon connection, and the places where the grant refs are mapped do not expect failure.
  • There are two ways to approach this:
    • Add the ability to set the refs array of the map by the client. This way, netfront can allocate its references and then set it up in the map.
    • Add the ability to pre-allocate grant references, and map them one-by-one. But for this, a mechanism to specify which reference is being mapped needs to be introduced. I need to talk with Roger about this.
  • One more essential thing I need to implement is the ability to load an already-loaded map. This is because netfront maps one grant ref at a time and creating a new map for each grant reference is overkill.

July 24-25:

Summary:

  • A lot of trial-and-error later, I figure out that the lowest value of gnttab_max_frames at which I can get the kernel to boot is 4.
  • But at 4 frames, the Xen DomU does not boot. The netfront driver prints “Failed to allocate tx refs” and then a page fault happens.
  • Tried to figure out a fix for the page fault, so I can submit the fix.
  • It originates from gnttab_free_grant_references() which is called from xn_setup_txqs(). The reason for the page fault is that earlier when the allocation of grant refs failed, the gref_head was set to GNTTAB_LIST_END. gnttab_free_grant_references() does not check for an invalid gref_head and thus causes a page fault.
  • Fixed that, but now there is a page fault in gnttab_end_foreign_access_ref(). This is caused because on fail, xn_setup_txqs() calls xn_disconnect_txq() which calls gnttab_end_foreign_access_ref(). But since there was a shortage of grant refs, the ring reference is GNTTAB_REF_INVALID and this causes a page fault.
  • Fixed that too, but now the kernel panics somewhere else.
  • I can’t spend too much time on this bug. I already have limited time with my college going on. I’ll let Roger know about this, and maybe he’ll figure something out.
  • I’ll try to test the busdma implementation in a shortage of grant refs, and then start working on netfront ASAP.

July 23:

Summary:

  • An email from Google arrived talking about how I should submit my work for the final evaluation. Will need to talk with my mentors about it.
  • Like Roger suggested, I started looking into creating a shortage of grant refs. When I set gnttab_max_frames=1 in the Xen command line, it does not even boot. Need to figure out the minimum number of frames I need to boot.
  • In the meanwhile, I’ll start organizing my code for the final evaluation.

July 22:

Summary:

  • College starts tomorrow. I won’t have as much time on my hands from now on. Still, I’ll try my best to finish as much work as I can before the official GSoC deadline.

July 19-20:

Commits:

Summary:

  • More testing. Tried multiple transfers to multiple disks at the same time. Works.
  • Review with Roger. He started reviewing the busdma implementation from the start once more. Check it here
  • Fixed multiple code style issues Roger suggested.
  • Wrote multiple lengthy explanations about some parts of the code Roger was asking about.

July 18

Commits:

Summary:

  • Before trying to run on a FreeBSD Dom0, I decided to attempt to narrow down which part is failing. So I started systematically.
  • First, I tried to determine whether or not we were even loading the memory properly. So, I reverted the blkfront driver to the original, and removed the code in the busdma implementation that handled the grant refs, and left only the code that handled the loading.
  • Compiled and ran. No errors. So the loads are happening just fine.
  • Then either the allocation/mapping of grant refs is going wrong, or the blkfront code is going wrong.
  • Tracing of every loaded map would not work. Most of the maps succeed, and there would be too many prints making it difficult to read the outputs, and the system would be slowed down considerably. To trace the code paths of only the maps that fail would require a per-map logging system. I could implement a simple per-map logging system, but I want to leave it as a last resort.
  • So, I next checked if the update to blkfront was wrong somewhere.
  • We have touched two areas: xbd_mksegarray()/xbd_queue_cb() and xbd_connect().
  • I reset xbd_mksegarray() to how it was before (this would mean double the number of grant refs would get allocated because the busdma implementation allocates its own, and then xbd_queue_cb() allocates its own. But let’s leave that for a while). The errors are still showing.
  • So, either the mapping of indirection refs is going wrong somewhere, or the busdma implementation is not properly allocating grant refs.
  • I reset xbd_connect() to the original and kept xbd_mksegarray()/xbd_queue_cb() to how it was post-update.
  • The errors disappear.
  • So, the culprit is somewhere in xbd_connect().
  • I finally have my eureka moment. I tried resetting xbd_connect() to the original before (I was not going for this systematic approach then), and the errors were showing up then. But this time they didn’t. It caught my attention that I forgot to reset block.h last time. So, I actually forgot to change the type of cm_indirectionrefs, which was an array in the original, and a pointer after the update.
  • Ah! There is the bug. When xbd_queue_cb() uses the indirection refs, it does a memcpy() to copy the grant refs of the indirection pages to the ring’s indirect_grefs. There it calls:
    memcpy(ring_req->indirect_grefs, &cm->cm_indirectionrefs,
    	    sizeof(grant_ref_t) * sc->xbd_max_request_indirectpages);
    
  • Notice the ampersand with cm->cm_indirectionrefs. It was needed when cm_indirectionrefs was an array. But now it is a pointer. Adding the ampersand would result in the wrong memory being copied.
  • Changing it to this fixes the errors.
    memcpy(ring_req->indirect_grefs, cm->cm_indirectionrefs,
    	sizeof(grant_ref_t) * sc->xbd_max_request_indirectpages);
    
  • A simple ampersand took me 4 days of debugging.
  • Tested the newfs command, like Roger suggested. Works.
  • Tested with low memory (512M) and copied large files, works fine.
  • I will start looking at the netfront driver tomorrow.

July 16 - 17

Commits:

Summary:

  • Lots of debugging.
  • The kernel boots, but it prints an error message every second or so.
  • Around 35% reads/writes have failed by the time the kernel boots and we log in.
  • The error is originating from an EIO returned by the ring request that we send to the backend.
  • Still haven’t figured out where or what is going wrong.
  • Asked Roger, he has no clue either.
  • Guess its trial and error from here on.
  • Looked at the backend code. There are 10 places where BLKIF_RSP_ERROR is returned. Defined the macro XBB_DEBUG, so DPRINTF() would print the error messages.
  • Weirdly enough, no error messages are printed by the backend.
  • Ah, realized my mistake. I am running on a Linux dom0, so making changes to the FreeBSD blkback won’t do any good.
  • I really don’t want to build the Linux kernel and hack it. I’ll try to run the domU on a FreeBSD dom0 and see if I can figure something out.

Monday 16 July 2018

[After First Evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

[After First Evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

For summary after the second evaluation, check the new post here

June 18 - 19:

Commits:

Summary:

  • Review with Roger.
  • He suggested multiple changes. Had discussions about some other issues.
  • Failing to allocate grant references should not be fatal. Find out a way to use the grant table free callbacks.
  • Removed the extra malloc tag I introduced.
  • The three _load functions have the same exact piece of code. Replaced it with a call to a helper.
  • Removed gref_head from the Xen DMA map. No need for it. Use gnttab_end_foreign_access_references() instead.
  • Realized a potential bug in the implementation. If the load is called with BUS_DMA_WAITOK and the load is deferred, the grant references were not getting allocated and loaded. This is because in case of error, we returned from without calling xen_load_helper(). So, when the allocation completed from a deferred context, the client callback was called with no grant references allocated. Fixed it (maybe) by passing a custom xen callback to the parent’s waitok().
  • Fixed it.
  • Failing to allocate grant references is no longer fatal. Using the free callbacks
  • First stipend arrived. Yay!

June 20 - 21:

Commits:

Summary:

  • Fixed multiple bugs and errors.
  • The client callback was not being called if the allocation of grant refs was deferred. Fixed it.
  • Fixed some memory leaks.
  • The code has gotten rather complicated after the introduction of xen_gnttab_free_callback(). Spent a lot of time reasoning about its correction.

June 22 - 23:

Commits:

Summary:

  • The code was getting rather complicated. So time to overhaul it a bit.
  • Roger suggested that I try to get the code in the same state before it reaches xen_gnttab_free_callback() so there is no need for that if.
  • I fixed it by calling the parent’s map_complete() a little early. This means that we can save a copy of the segs array in xen_load_helper(), and the free callback has to simply use the copy in both cases: initial load deferred, and not deferred.
  • We need to call the parent’s map_complete() anyway, so it’s not like we are doing anything wrong. This is because once the parent’s load is complete, its map_complete() should be called. But if the allocation of grant refs fails, we will sleep. It is better to complete the parent’s load cycle before waiting.

June 25:

Commits:

Summary:

  • Roger had a great idea about the problem with moving more of the duplicated code to a single place. Since the loads use different types and number of arguments, he suggested that I use a union.
  • Modified the loads to use the union and moved the duplicated code to xen_load_helper().
  • Roger also pointed out that the code would be unreachable because I was doing:
    if (error == EINPROGRESS) {
    	return (error);
    }
    else {
    	return (error);
    }
    
  • But I should instead have else if (error != 0) otherwise the code below would never run.

June 26 - 27:

Summary:

  • The Xen-specific bus_dma(9) implementation is almost complete, other than some minor additions/fixes:
    • Memory leak and segmentation fault if unload is called before the load finishes.
    • The ability to reserve a pool of grant references, like netfront and blkfront do.
    • The ability to extend an already loaded map.
  • Wrote the freebsd-xen@ and freebsd-arch@ mailing lists to ask if I should expect a map to be unloaded before the load completes. No responses so far.
  • Looked at the netfront driver to understand how it works and convert it to use the bus_dma(9) implementation we wrote.
  • Need to write tests for our implementation. Spent some time reading sys/dev/xen/netback/netback_unit_tests.c.
  • Wrote to my Edward, Roger and Akshay about the tests. Roger says it is too much work, and testing of a busdma implementation has never been done before. It is probably a GSoC on its own.
  • So scrap the plans about testing.
  • Roger suggests I start with blkfront.
  • Started reading through the blkfront code.

June 28-30:

Commits:

Summary:

  • Lots of reading. Still going through the blkfront code.
  • Did some grepping around to find how other drivers use bus_get_dma_tag().
  • Not much success, there are a lot of results, and most don’t do anything interesting, they simple call the parent’s bus_get_dma_tag().
  • Looked at how busdma_dmar does things.
  • They have a function dmar_get_bus_dma_tag() that returns the ctx tag after some initialization.
  • Looked into sys/x86/iommu/intel_ctx.c. They simply set the parameters to max.
  • Also looked at sys/dev/acpica/acpi_pci.c. They use the dmar tag.
  • Wrote xen_get_dma_tag() that can be used to get the xen-specific dma tag.
  • Added xenpv_get_dma_tag(). It can be used by devices hanging off the xenpv bus to get the xen dma tag.

July 2-3:

Summary:

  • Read an article about the Xen device drivers.
  • Lots of code reading.
  • Made some minor changes/fixes to xenpv.c

July 4-5:

Commits:

Summary:

  • Started converting the blkfront driver to use the Xen-specific bus_dma(9) implementation.
  • There are two places grant refs are allocated or foreign access granted in the blkfront driver:
    • xbd_connect(): Here, foreign access for the xbd command indirection pages in granted.
    • xbd_mksegarray(): This function is called from the callback specified to the dma load called from xbd_queue_request(). The callbacks calls xbd_mksegarray(). This function more or less does what our busdma implementation does: get ds_addr and grant foreign access to that address. Converting it would be tougher than converting xbd_connect() because it chases a lot of functions and is more complex.
  • Converted the xbd command indirection pages to use the busdma implementation.
  • Will ask Roger for review, and then start converting the dma loads in xbd_queue_request().

July 6:

Commits:

Summary:

  • Read a blog post about Xen block device indirect pages.
  • Review with Roger.
  • He suggested some small changes. Did them.
  • Discovered a bug in xen_bus_dma_tag_create() which would cause an infinite recursion if a driver creates a tag. This would happen because we call bus_dma_tag_create(parent, ...) from xen_bus_dma_tag_create(). The call to tag create would expand to parent->impl->tag_create(parent,...). Parent’s tag_create is xen_bus_dma_tag_create(), so this would result in an infinite recursion. Need to fix it.

July 9:

Commits:

Summary:

  • Fixed the bug in xen_bus_dma_tag_create() which would cause an infinite recursion if a driver creates a tag. Achieved this by passing parent to common_bus_dma_tag_create() instead, and pass the parent tag’s machine dependent tag to bus_dma_tag_create().
  • This raises another problem: xen_get_dma_tag() calls xen_bus_dma_tag_create() with the machine dependent dma tag in the parent argument. So there is need for a way to signal to xen_bus_dma_tag_create() that this is a special case where we are “bootstrapping” the dma tag. Used BUS_DMA_BUS1 flag here, which is reserved for the busdma functions to use as they wish.
  • Added a refcount check in xen_bus_dma_tag_destroy(), so that we don’t free an in-use tag.

July 10-11:

Commits:

Summary:

  • Filled the second GSoC evaluation.
  • Multiple minor fixes like disabling indirection pages if the load for them fails, shifting ds_addr by PAGE_SHIFT.
  • Started updating xbd_mksegarray() to use the busdma implementation.
  • There is a problem: xbd_mksegarray() uses ds_addr, but we replace it with the grant reference, so xbd_mksegarray() will not work properly without it.
  • Introduced the function xen_dmamap_get_grefs() that client drivers can call to access the grant references. This makes it possible to access ds_addr and the grant refs simultaneously.
  • Updated xbd_mksegarray(), xbd_queue_request(), and xbd_int() to use the busdma implementation.
  • The blkfront driver is completely updated to use the Xen-specific busdma implementation now. What remains is to, of course, test it.
  • I need to build the world and kernel, and it takes a really long time. I’ll probably leave it running overnight.
  • Also need to figure out how to convert the source build into an ISO so I can run it as a guest.

July 12-13:

Summary:

  • Time to test the code I wrote.
  • Roger suggests I grab one of the pre-built VM images from the FreeBSD FTP server and run the build in them.
  • Downloaded the image and tried to boot. The boot gets stuck somewhere in the early stages.
  • After a lot of looking around, I decide to try the disk image in RAW format and not in VHD format.
  • Voila! It boots.
  • Now I need to set up networking so I can fetch the FreeBSD sources and build them.
  • Turns out it is much tougher than I expected.
  • The official Xen guide does not work. The VM detects xenbr0 but it does not correctly set up the network. I can’t SSH into the VM from dom0 or connect to the internet.
  • Googled around more. No success.
  • I caught a lucky break. As a last resort, I try the lxcbr0 bridge I set up for a VirtualBox VM some time ago, and I had it lying around in my network interfaces. It works!
  • After some tinkering around with the DNS settings, networking finally works in the VM.
  • Checked out the code from my repo and compiled.
  • The kernel panics on boot.
  • The panic comes from a failed assertion inxbd_indirectpage_cb(). After some looking around, I realise the assertion condition is wrong. I was using sc->xbd_max_request_segments and I should instead have used sc->xbd_max_indirectpages.
  • Fixed it.
  • The kernel now boots. But it keeps throwing some error messages every few seconds. Looked around, but couldn’t fix it. Need to ask Roger about it.

For summary after the second evaluation, check the new post here

Friday 11 May 2018

[Up to First Evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

Note: This post has gotten rather long. For summary of everything after the first evaluation, check the new post here.


Up to May 10:

I did a lot of stuff before deciding to publish a progress blog. I will try to list as much here as I remember:
  • Set up FreeBSD in a VirtualBox VM.
  • Checkout the FreeBSD HEAD and build from source.
  • Checkout OpenBSD's repo.
  • Read through OpenBSD's Xen implementation and try to make sense of the code.
  • Learn about Grant Tables and Grant References.
  • Read about what DMA is and how the bus_dma(9) interface works.
  • Try to understand how OpenBSD uses the bus_dma(9) interface in its Xen driver.
  • Listen to a talk by Mike Belopuhov, the person who wrote the Xen drivers for OpenBSD, to learn more about the implementation
  • Listen to a talk by Stefan Sperling about developing device drivers for OpenBSD to get a better idea of how drivers work in OpenBSD, and thus getting some idea of how the Xen driver will work in the system.
  • Subscribed to some mailing lists which might come in useful for my project:
    • freebsd-hackers
    • freebsd-virtualization
    • xen-freebsd
    • openbsd-tech
    • xen-devel (xenproject.org)
  • Tried to understand where bus_dma(9) will fit in FreeBSD's implementation (not much success. I'll ask my mentors a few questions about it).
  • Created a wiki page about my project.

May 11:

  • Created this blog and wrote my progress so far.
  • Time to finish setting up my working environment. Talked to my mentors about it. The problem was how to move code from my Ubuntu working machine to my FreeBSD testing machine. Edward suggested sshfs, Akshay suggested scp. I'll try both
  • Currently, my VM is building world. It is taking a long time.
  • In the meantime, I'll learn about ssh and scp and see how I can set them up. Setting up a bridged network from my VM to host looks like a good idea. If this does not work, I'll try doing it over the internet. Not sure how either will work though, I need to wait for the world to finish building.
  • Read about how to use SSH and how SSH works.

May 12:

  • Read up about how to configure the SSH Daemon.
  • After a lot of Googling, finally found out how to connect my host to my VM via SSH. Turns out using a bridged network on my Wi-Fi interface was the answer. I spent a lot of time trying to forward my port 22 to my VM over a NAT network, but that didn't work.
  • Finally able to SSH into my VM.
  • Tried out scp, it works.
  • Got git-scp to work. Now I can transfer changed code from my working machine to my testing VM.
  • git-scp has a weird bug where it stages my changes after sending them over to the remote machine. Need to look into that.

May 14:

  • Changed from password auth to RSA key auth in SSH.
  • Enable the serial console in the FreeBSD VM to get kernel logs. The serial port can be configured in VirtualBox settings to output to a file.
  • Time to install Xen and get a DomU running.
  • Strangely, the Xen package is missing from the FreeBSD package repository. Maybe I can build from source.
  • SVN dropping the connection during checkout. Need to check my internet connection.
  • Couldn't get SVN to work. So used a hack. The packages missing were there in FreeBSD 11's repository. Downloaded them from there and installed. Xen is now installed.
  • The system does not boot. It is stuck in a loop. It goes from boot screen (select whether to boot multi user, single user, etc) and reboots to there again. Maybe my hack was not a good idea.
  • The system went into a boot loop because of a typo in /boot/loader.conf. Fixed it through a Live CD.
  • The kernel panics and does not boot. Xen writes an error message saying Intel processor family 6 processor 60 not supported. It looks like Xen might not boot in a VM.
  • Wrote to freebsd-xen mailing list about the problem. Let's see what they say.

May 15:

  • So it turns out we can't boot Xen with FreeBSD as Dom0 in VirtualBox. Unless I can find another hypervisor that can run this, I will have to boot FreeBSD on my old laptop.
  • Spent a lot of time trying to fix the boot loop in the VM to little success. I am now dropping the idea to use a VM for testing.

May 16-17:

I spent a LOT of time trying to get Xen to boot on my other laptop to no success. These two days have been very frustrating and tiring. I feel like I'm wasting so much time just trying to get Xen to boot. I got Xen to boot on Linux in a few minutes. Getting it to boot on FreeBSD has been a massive pain so far.
  • Time to install FreeBSD on my old laptop. I want to keep my Windows so I'll try to install FreeBSD on a different partition. Then I can restore Windows bootloader when I'm done with the project.
  • Had to drop the idea of keeping Windows. FreeBSD complains about the disk being MBR not GPT and it can not install in GPT.
  • Format the disk and install FreeBSD. Set up the environment, install some useful packages and get my network card working.
  • Time to install Xen.
  • Xen installed. On reboot it says don't know how to load module.
  • Googled around a bit. Turns out I installed FreeBSD in UEFI mode and Xen can't boot in UEFI.
  • Here is where I hit my biggest roadblock. I can't figure out how I can install in Legacy mode.
  • After hours of reading through forums and mailing lists, I finally figure out why I can't install FreeBSD in legacy mode. There in a bug in my laptop's BIOS where it does not detect USBs with GPT in Legacy mode.
  • My working laptop does though, so I use it to install FreeBSD on MBR-formatted USB.
  • The USB boots in my old laptop.
  • At this point a hack comes into my mind. I boot a Live CD from another USB drive and use dd to copy the contents of the USB disk with FreeBSD to the hard disk of the laptop.
  • Works! Now I can boot FreeBSD in Legacy mode on my old laptop.
  • Let's install Xen. Ugh... Doesn't boot.
  • Ask on the freebsd-xen mailing list. Not much help.
  • So finally I decide to try it on my new laptop. Create a USB, insert it and boot FreeBSD.
  • Installed Xen, it goes further than last time but then gets stuck on an empty screen. Ask on the freebsd-xen mailing list.

May 18:

  • Akshay's reply puts me on the right track. After a bit of Googling, I fix that blank screen.
  • Ugh... STILL DOES NOT BOOT. I'm getting really frustrated now.
  • Fiddled with the Xen command line, no change.
  • Wrote to the freebsd-xen mailing list about it.
  • After a bit more research, I find that my old laptop's processor does support VT-d but there is no option to enable it in the BIOS; just an option for VT-x. So no way I'm using the old one for testing. My new laptop though, does support VT-d. I will use it for testing. But right now, I can't get it to boot. No response from the mailing list yet.
  • There is a weekend coming, so I might not get a response till Monday. I'll spend the Saturday reading through some grant table code. I feel like I spent too much time on trying to set up my testing environment. Need to compensate for that.

May 19:

  • Akshay suggested I should try booting from my hard disk drive (HDD) rather than my USB.
  • Started looking in using GRUB with a mix of BIOS and UEFI OSes because I don't want to lose my Windows and Ubuntu installations. Not much success, but I did find one StackExchange answer that answers exactly my problem. I need to create a partition and label it with bios_grub.
  • Created the partition and installed FreeBSD. Does not boot.
  • Hm, probably something wrong with my GRUB config. Open a Linux Live CD to install grub. Can't install because GRUB does not recognize FreeBSD. Also, the FreeBSD partitions do not mount, and I need to save the config in /boot/grub/.
  • Oh! I never realized GRUB also had a package for FreeBSD. Time to try it.
  • Chrooted into my installed system using a FreeBSD Live CD. pkg needs to be bootstrapped, and it fails.
  • A hack comes in my mind. I modify the /etc/fstab of my Live CD to point root to an invalid location. The root mount at boot will ask me to specify the device of the root filesystem, and I can give my HDD location there. So basically I use the Live CD for bootstrapping and then mount my HDD system.
  • In the HDD, install grub. Works. Then used grub-mkconfig to create the config file. Reboot. Does not boot.
  • Again used the hack to get in my HDD installation. This time, I hand-write my grub.cfg that I found on the FreeBSD forums. Reboot. We get into grub!
  • But FreeBSD panics on boot. Googled some more config files for FreeBSD and noticed that some people are specifying kfreebsd /boot/loader, but my config is specifying /boot/kernel/kernel. Fix it.
  • Works! I can now boot FreeBSD from my HDD.
  • Installed Xen and guess what, it doesn't boot. Same panic message as when booting from USB. All I did to get GRUB to work was an exercise in futility.
  • Roger replied. He suggested I try dom0_max_vcpus=1. Also asked me to give some info for debugging.
  • FINALLY XEN BOOTS! YAY!

May 21-22:

I don't have much to report for these two days. It has been a lot of reading, and there isn't much material to write here.
  • I still have not identified where the bus_dma(9) wrappers should go in FreeBSD's Xen implementation. I am much more familiar with the OpenBSD implementation than I am with the FreeBSD one. I need to ask my mentors about that, but I feel I should do a bit more homework before asking. I'll draft an email by tomorrow evening.
  • Listened to Mike Belopuhov's talk on OpenBSD's Xen implementation again. I understand it much better now than I did before.
  • He mentions the Xen Netfront drivers need to use the grant tables, and he explains how the bus_dma(9) can be used for that. Started reading through FreeBSD's Netfront driver. It's huge.
  • Started reading OpenBSD's Netfront driver instead. It seems much simpler, in part thanks to Mike's talk.
  • Every time I try to understand FreeBSD's implementation I just feel lost. Need to fix that. I do have a little better understanding of the OpenBSD implementation, mostly because it's simpler and I have spent more time understanding it.
  • At this point, I feel like I understand how grant tables work, and how they use bus_dma(9), but I do not have a grasp of the bigger picture; about how all these systems combine together. Will ask my mentors/Roger about this.

May 23:

  • More reading. Read the OpenBSD Xen Netfront driver (sys/dev/pv/if_xnf.c) to understand how it uses the bus_dma(9) wrappers. It calls bus_dmamap_load() and then uses the ds_addr from the dma_seg_t which contains the grant reference of the page. That grant reference was loaded by the custom wrapper method xen_bus_dmamap_load()
  • Looked at the FreeBSD man page of bus_dma(9) interface. It is quite different from OpenBSD's interface. I did not expect them to differ much. It seems like implementing the wrappers is more work than I thought.
  • Reading the man page, I do think how I might achieve the same effect as OpenBSD. OpenBSD's interface seems more intuitive.
  • Rather than creating a custom bus_dma_tag I need to use the callback functions to call the xen-specific methods and update the grant table from there.

May 24-25:


  • Decisions to make. Emailed Edward and Akshay about the potential design choice we have to make. Haven't heard from them yet.
  • Dug deeper into FreeBSD's bus_dma(9) interface. I realize there are more things to take care of than I thought. The different interface OpenBSD and FreeBSD's bus_dma API exposes is posing some problems.
  • Wrote the xen_bus_dmamap_load(). More problems present themselves. It turns out I have to use the callback method. This creates the problem of returning error codes in case allocating a grant reference fails.
  • Maybe I can allocate the grant reference when we create the bus_dma_tag with frame number 0 and then simply update the frame number in the callback. This would mean we won't have to deal with returning error codes from the callback method.
  • Pushed a commit of a rough implementation of xen_bus_dmamap_load(). Check it here.
  • Unrelated: I noticed there was some code in gnttab.h that was wrapped in a #if 0. Asked Roger about it, he says it is left over from when it was imported from Linux. Submitted a patch to delete it. Roger reviewed and accepted it. Check it here.
  • Pushed a commit of a rough implementation of xen_bus_dma_tag_create(). Check it here.
  • Pushed another commit updating xen_bus_dmamap_load(). Check it here.

May 26:

Today's commits:
  • Still haven't heard from my mentors. In the meantime, a rough implementation of all the main functions is done. I still have not done bus_dmamap_create/destroy() and bus_dmamap_load_mbuf. The dma map creation/destruction methods are pretty straight forward. I'll probably call the main bus dma methods and return straight away. No grant table related stuff needs to be done there.
  • The mbuf variant of bus_dmamap_load has to be done, but this should be pretty similar to the non-mbuf variant xen_bus_dmamap_load.
  • This leaves us the review of design choices and code review. I'll wait to hear back from my mentors. There isn't much I can do right now.

May 28-29:

Commits:
  • Asked my mentors for review of my implementation. Haven't heard back anything yet. I have not received replies to my emails for a few days now. Maybe they are busy at work. There isn't much I can do right now until I get some reviews.
  • Fixed some minor problems.
  • Fixed the issue of not being able to call the client driver's callback function. Used a struct containing the function pointer, client's callback argument and xen's callback argument. Pass this to the xen-specific callback we give to bus_dmamap_load() and call the callback from there.
  • Compiled the code to catch syntax errors. Fixed the errors found.

May 30:

Commits:

May 31 - June 2:

  • I was on holiday these 3 days.

June 4-5:

Commits:
Summary:
  • Asked Roger Pau Monne to review my implementation.
  • He pointed out a bunch of code style problems. Fixed them.
  • He also pointed out a potential security flaw where every domain would have access to physical frame 0. Will fix it in a follow-up commit.
  • So it turns out I misunderstood what exactly I had to do. I built the Xen-specific bus_dma(9) handlers on top of the existing bus_dma(9) interface. Instead, I was supposed to build another implementation of the bus_dma(9) interface specific to Xen, kind of like how busdma_dmar is implemented.
  • I'll have to rewrite some code, and add some other handlers to create the alternative bus_dma(9) implementation. I will start on it tomorrow.

June 6:

Commits:
  • Need to refactor the functions I wrote to conform to the bus_dma implementation headers exactly.
  • Introduced the bus_dma_tag_xen. It will contain the information needed for the xen-specific dma 
  • Added more fields to xen_callback_arg. These will be used for grant allocation in the callback.
  • Converted map load, map unload, tag create, tag destroy.

June 7-8:

Commits:


Summary:
  • A lot of reading. Looked at busdma_dmar.c and busdma_machdep.c and busdma_bounce.c to see how other variations of the bus_dma(9) interface are implemented.
  • I'm confused. There are so many things to factor in.
  • Let's take it step by step. Wrote tag creatiion and destruction. Now time to start with map creation and destruction. Looks simple enough. The different types of load (load_phys, load_buffer, load_ma) are the problem. I need to figure out how to handle each.
  • Finished writing the new implementation. Need to compile it and go through it again before asking for a review from Roger.

June 11:

Summary:
  • Code review with Roger Pau Monne. He did point out some small mistakes, but nothing major.
  • Came up with a plan of what to do when a grant allocation fails. Earlier, the errors were silent. Roger suggests we handle the errors during map_load_*(), and not during map_complete(). Sounds like a good idea to me.
  • Discussed the issue of the parent tag allocation. I was creating a tag descended from the parent tag, which was then saved and later used for the dma operations. Roger says that is not necessary. The parent can not be destroyed as long as there is a child for it, so no need to create another tag. Just save the parent directly.
  • First GSoC evaluations open today. It will be 9:30 PM in my time zone (IST) so I'll probably fill them tomorrow.

June 12:

Commits:
  • Filled the GSoC First Evaluation.
  • Made the changes that Roger suggested.
  • The grant references are now allocated in tag create, so that there can be no failures in map_complete. Refactored the code accordingly.
  • Defined a new malloc type M_BUSDMA_XEN and used it to allocate the refs array, instead of using M_DEVBUF.
  • Earlier, map_complete did not check for the error code and made the allocation anyway. Now, it checks the code and does not map the references if there has been an error.

June 13:

Summary:
  • Asked Roger for this thoughts about yesterday's changes. He pointed out one problem. I am allocating the grant references in tag create. But what I didn't know was that a single tag can be used to create multiple maps. So the grant references should be allocated for each map, not each tag.
  • This means a major refactor. I was not using a Xen-specific dma map before. But now I need it because it has to hold the grant references array and other associated data.
  • All the hooks that use a dma map need to be refactored.
  • Done with most of the changes. Will finish up tomorrow.

June 14:

Commits:
  • Looking at busdma_bounce.c, I see that a map does not necessarily use all the segments. Dive deeper into the implementation to see if there is any way to find out how many segments it does use, so we can allocate as few grant references as we can.
  • Further investigation leads me to suspect there is a bug in the busdma interface. Wrote a long email to freebsd-hackers@freebsd.org about it. Check it out here.
  • So I did find out a way to get the number of segments a map uses. It looks kind of like a hack to me. Not sure whether I should use it. I'll ask Roger.
  • Finished up the changes.

Note: This post has gotten rather long. For summary of everything after the first evaluation, check the new post here.

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD The links to the commits are prob...